Unfortunately, the answer is yes, holes in the lower trunk do indicated borers.
The most common borers for these trees are the flat headed apple tree borer and the round headed borer.
The rounded headed borer is the larva of a particular type of beetle. If this is the case you should see the beetle around this time of year (April/May) as it begins to lay its eggs.
To control borers check the tree carefully and cut and burn any dying stems below the holes. In around June, you can crush any eggs you see on the bark. In the summer you should see fresh bark wounds beginning to form. It is best to cut the holes out with I knife as soon as they begin to appear. If the holes are straight it is possible to kill the borers by using some type of wire to clean out the hole and destroy the larva. You can also use nicotine sulfate to kill the borers. This can be done by dipping a cotton ball into 1 part solution and 4 parts water and placing them as deep into the holes as possible.
The good news is you should be able to get rid of the borers but it may take some time and effort.