This process should work regardless of the order the motor phases are plugged in to the controller, or how the position sensor is initially oriented.
Step 1: Determine Phase Ordering
The purpose of this step is so that commanding positive current on the q-axis produces torque in the direction that causes the encoder angle to increase. Basically, the process is to apply a large, slowly rotating current to a "virtual" D-axis. The rotor will closely follow this rotation. This is basically equivalent to driving the motor like a stepper motor with microstepping. As the current rotates around, if the encoder count is increasing in the positive direction, then everything's good. If the encoder count is decreasing, then swap the voltage outputs and current sensor inputs on two phases. Now the motor will rotate the correct direction.
Step 2: Measure Encoder Offset
Now that the motor spins the right direction, you can measure the DC offset of the encoder. Just like before, apply volts to the D axis, and slowly rotate the axis through a whole mechanical rotation both backwards and forwards.
Looking at the position sensor output vs the reference angle, you'll see something like this.
Zooming in a bit, you can see some ripple in the tracking from cogging torque:
Looking at the error between the reference angle and the actual rotor angle, things get a little bit more confusing. The error plot below has a few interesting features.
First, and most obviously, the mechanical offset of the encoder is just the average value of the error. The electrical offset (which is what we actually care about for commutation) is just mod((mechanical error) * (number of pole pairs) , 2π).
The high frequency ripple (which in this zoomed-out view looks almost like noise), is from the cogging torque (the ripple that showed up in the red trace above).
Right in the middle of the plot, there's a jump downwards. This is where the motor switches direction. Since the motor has friction and inertia, it always lags slightly behind the reference angle, so that lag switches sides when the motor switches directions.
Then there's some low-frequency, much larger ripple on top of the signal. This is from eccentricity of the position sensing IC/its magnet. In my case, it's because I accidentally messed up the design of a PCB and placed the IC 0.5 mm off to one side. The total peak-to-peak height of this ripple is only around 0.03 radians, or 1.7 degrees, which doesn't sound like a lot. However, I'm using a 21 pole-pair motor, which means that 1.7 degrees of mechanical error translates to 36 degrees of electrical error. Enough to seriously throw off commutation. The next step will go through how I corrected for the error from eccentricity. While this won't be nearly as much of a problem once I get the new round of boards in, there will always be a little error in the placement of the IC (especially when they're hand-soldered by me), so having this feature is still definitely useful.
Step 3: Eccentricity Correction
The first trick for correcting for the eccentricity is separating out the error from cogging and friction from the eccentricity error.
Friction is easy. Just average the samples rotating forwards with the samples rotating backwards. Then plotting error you have something like this
Removing the ripple from cogging torque took some more thought, but it turns out can be done extremely effectively. My initial though was to just low-pass filter the signal, and hope that cogging could be sufficiently attenuated without changing the eccentricity part of the signal too much. Honestly, for this particular motor with lots of pole-pairs and many cogging steps per electrical cycle, that would have probably worked just fine. But you can do better, and make the result more easily applicable to other motors.
The trick is to take advantage some properties of motors and FIR filters. Since the motor is rotationally symmetric, the frequency of the cogging ripple must be some integer multiple of the electrical frequency. FIR filters can be designed such that they have zero gain at certain frequencies and harmonics of those frequencies. So you can easily design a filter that has zero gain at the electrical frequency, and all multiples of it, nearly perfectly cancelling out the cogging ripple without affecting lower frequencies. I'm using basically the simplest possible FIR filter - I just average n samples around the point of interest, and choose n such that the samples I'm averaging exactly span one electrical cycle. This should work pretty well on any motor, as long as the number of pole pairs is larger than the highest harmonic from eccentricity. Here's what the the filtered version of the signal looks like:
Now, with that filtered signal, you can build a lookup-table to correct the encoder output. It doesn't even need to be particularly high-resolution, since the signal varies slowly - I'm using 128 points, and it works great. The lookup table is generated by subtracting the average value off of the error, and then converting the error back into raw encoder counts. I'm cheating a little bit and using the reference position as the x-axis, rather than the actual encoder value, because it's nice and evenly spaced into a grid already. This approximation works quite well though.
Here are three lookup tables generated independently, showing the consistency of this technique:
My latest round of motor control firmware can be found here, if you want to look through the autocalibration code. It's fairly unpleasant to read, but it is well commented. The code as a whole is very much a work in progress, so don't expect to put it on anything and have it work out-of-the-box.
At some point I'll throw a high-resolution optical encoder on the output to quantify the improvement in position error, but in terms of torque ripple, this process qualitatively improved things enormously.