You Certainly can! But like anything in the building game there are always a few things to think about before you build anything on the boundary.
One of the main important things to consider is the development act of 1993, it states that you must consider future work to the property next door and allow in the design an allowance of future work below the retaining wall to a depth of 600mm. This can be quite easy to work out, if you are in a developed area and the properties have been there for years, you are probably not going to need to stress about it (although still seek it out) as the likelihood of your neighbor to cut out will be rare seeing that his land is already established.
But if you are building in a new development like a new sub division and you are the first house to cut and fill and to build your house, you will need to think about this.
I have seen so many times people getting impatient and building their walls before their neighbors have even developed their block, what can happen is the the new wall gets undermined and looks terrible and can likely fail.
Sometimes its best to be patient to get a nice job
But if you cant wait and you want to move into your house and you have young kids or dogs and you want your property to be secure it will be in your best interest to follow this act, as it makes allowances for the pier and footing depth of your retaining wall, so if your neighbor does cut out it wont be the end of the world, although it will be annoying because your neighbor will have the option to build in front of your wall to hide the gap but he may want to jackhammer the concrete footing and simply slide in more concrete sleepers, i say simple but its actually hard work!!
But it can be done.
Always check with council first
Check with your local council first, because in some states you cannot build a retaining wall closer than 600mm to the boundary and in other states thats 900mm!
Do You know where your boundary is?
If you dont know where the boundary is it might be a good idea to dust off your old house plans to know where they are or if you dont have them and want something more accurate get a surveyor out to mark out the boundary points with steel or timer pegs, that way you wont have any squabbles down the track with your neighbors.
'The regulations are there to protect you and the neighbor'
As annoying as council is and the fluffing around they do the regulations and approvals are here to help you and your neighbor have the retaining wall in the right place and to make sure its built properly and doesn't fail!
Things to consider:
1. If you are installing a retaining wall inside your property ensure that in doing so doesn't affect the adjoining property
2. Make sure that your wall has correct drainage so it doesn't impact your own yard or runoff into your neighbors yard.
3. That the retaining wall you build inside your property is still built to standards
If you and your neighbor agree to build a retaining wall on the boundary make sure you go to council first, i cant stress this enough, i've seen it happen too many times when people don't do things properly and it can cause a lot of stress between both parties.