Ok everyone – we’re back with another blog, lots of tech to talk about. Grab you popcorn, have a seat, and lets get started.
These past few weeks have been full of many tough challenges– re-vamping our manufacturing process, fixing our drive quality control system, and automating our labour intensive quality control and calibration processes.
It’s reminiscent of the day back this fall when we had to create a second closed loop system. We knew there was a way, but the question became a matter of practicality – mainly focused around time constraints, cost constraints and simplicity of the solution.
But once again it comes down to the amazing people working on the Mosaic team. In the past two weeks we’ve gone through multiple iterations of our drive QC system, as well as completely changing up our manufacturing process and cutting our QC and calibration time down by 50%.
These are the three main areas we’ll break this update down into: Streamlined Manufacturing Processes, Drive QC testing, and QC/Calibration Automation. In each of these areas we’ll break down the problems we were having, the process we went through to identify and change these problems, and the outcome of each of them.
Process Changes
Back in April we were producing between 8 and 10 Palettes each week. Our first batch had an 80% fail rate, our second batch had a 50% fail rate. This means that in our first week we shipped two Palettes, and our second week we shipped five. This comes out to (roughly) 15 Palettes/month. At that rate, it would have taken us over one year to fulfill our Kickstarter orders alone.
When you look at the situation that way, you begin to understand that serious changes needed to be made….. so that’s what we did.
Before, each assembly team member completed a variety of unrelated tasks. One person would be working on soldering in the morning, and completing drive assembly in the afternoon. We had a set number of tasks that needed to be completed in a given week, and we moved through them accordingly.
The biggest issues with this approach were a lack of specialization, lack of goals, lack of measurability, and a massive lack of efficiency.
So those are the four things we changed. Each assembly team member now focuses on a specific sub-assembly (or QC/calibration), tracks how much time each task takes them, records issues they run into which slow them down, and has a set number of sub-assemblies they have to build at specific check in points during the week.
Below is an example of a weekly schedule – this one covers our electronics tray.
In any given week, the team member responsible for this cell must hit the “LT” (which, in English, means 25 units).
Now, what we have is a schedule to hit a specific number of Palettes (25) in a given week.
Check out the results of the process changes below:
In the above picture you can see our in-process QC and Calibration Wall. 30 Palettes in this picture. (sorry for the poor lighting!)
Here is our QC overflow wall. 7 Palettes in this picture.
Here is the shelf for the final assembly step. These Palettes are waiting for their electronics trays before moving out to the main room (QC/Calibration shelves pictured above). 15 Palettes in this picture.
Here you can see more of our in-process Palettes. These Palettes are waiting for drives, and electronics (plus a few smaller items) before they’re ready to move out to the main area. ~19 Palettes in this picture
We decided to take advantage of MakerBot’s special Amazon pricing and grab ourselves a few more Rep 2s. These are being added for print test purposes which will allow us to increase our throughput at this stage.
All in all, these process changes have been a big success so far. Because everything is being tracked at a deeper level, we’ve been able to identify what is slowing our team down and get them the appropriate tools (or process changes) to make everything more efficient.
We will be aiming to get up to 30 per week in the coming months, as improvements continue to be made to these processes.
Drive Quality Control
The quality control of our drives is the main cause of our QC and print test fails. A lot of what we do is based on the manipulation, control and movement of filament – drives are essential to this.
There are five drives in the Palette, and if any one of these five is not operating in its optimal state the Palette will give unacceptable results. When we have splice issues, it’s usually due to drive accuracy. When we have broken splices, the root cause is drive accuracy. When we have loss of calibration, that’s usually due to drive issues as well (or, to be more precise, the pairing of drives to a given scroll wheel).
There are a few different failure modes for drives: slipping or skipping.
Skipping is probably something you’re familiar with on your 3D printer. It’s when the filament can’t move forward for whatever reason. Usually with a 3D printer it’s due to a jammed extruder – but for us, skipping is rarely an issue. The only time skipping is a problem is if the filament curve causes the filament to hit something inside the Palette. This can happen in the splicer Teflon area, and sometimes in the outgoing drive. This skipping, however, is not a drive issue as it is caused by filament occasionally getting caught on something inside the Palette during its startup procedure. After the start up however, it is very rare that filament will ever get caught on anything
Slipping is where our issues are. Basically, slipping is the opposite of skipping. Skipping there is too much pressure on the filament. With slipping there is not enough. We believe that the majority of our slipping issues come from the distance between the roller bearing and the drive gear. We believe the main reason this slipping was not caught is because the drives were not being properly worn in before our in circuit QC tests were completed.
Now we’ll take you through what we were doing, our iterations, where the system is today, and what the results are.
And just because it’s a fun side-by side, here is a picture of our first QC Rig:
Here is a picture of our current rig:
Our first rig (paint can system) was a way to measure whether or not the drives could pull filament with a significant amount of resistance on it. This rig did what it should have (measuring if a drive can pull), but we discovered after a few weeks that it was seriously flawed in a few main ways. We’re still working to discover exactly why this is, but there is a factor of things “working themselves in” on the Palette. This is particularly true for the drives and drive gears.
Initially, with the paint can system, we would put filament through the drives, run it, and if the drives pulled a certain distance they passed the QC test. However, sometimes these same drives would fail when they were completing a print test, or another step in the QC process.
One new step in our new process is to run 1-2m of filament through the drives before they are tested. This is to allow everything time to settle in, so we can receive more accurate results when the tests are run.
The two drives you see on the top of the picture (in the blue jig) serve as resistance points for the filament. The two drives you see in the middle (orange jig) are the drives being tested, and the two Scroll Wheels you see in the bottom (blue jig) are measuring the amount of filament each drive is pushing.
Measuring Filament
When we run our QC process (covered later in this post) we receive an input (scroll wheel reading) that we put into our system to help us calculate steps per count. Steps per count is the number of stepper motor steps in each tick (count) on the scroll wheel. This is one of the calibration numbers needed to effectively run your Palette. We run two tests per drive, record the numbers, and input them into our system.
As we explained in our last update, we noticed something interesting in our drive numbers when we looked into which Palettes were failing. There are two main areas we began investigating, both of which we noticed what could be the beginning of a trend.
Test 1 vs test 2 (in a single drive).
The two numbers from these tests are usually in a range of 14200-14400. If the drive we tested had a difference of greater than 100 between these values, the Palette (almost) always failed in the later stages. This was true for 9 out of 10 machines.
Learning from this, our new rig runs two tests, and if the discrepancy between the two tests is too high, the drive will go in the fail bin and be parted out for another test at a later date.
We believe that these inconsistencies may be caused my minor slipping in the filament, on a minute enough level that it is not detectable by an individual.
Drive 1 vs. Drive 2 vs. Drive 3 vs. Drive 4.
The other important number in this system is the similarity of readings between every drive in a given Palette. For example, if drive 1 gave readings around 14,100 and drive 3 gave a reading around 14,400 the difference would be 300. We noticed a very high failure rate in Palettes that had differences of 200 or more between their drives.
We believe this drive mismatch can lead to imperfections in splices. Splicing is a precise art, and the control algorithm for the filament is finely tuned. The results of this algorithm are thrown off when two drives are driving filament different distances.
So, we added a second test completed with new rig that batches drives into certain groups based on how much filament they push during their tests. This test also tells the operator whether or not the two drives being tested are within an acceptable range. If they are not, one of the two motors is replaced and the test is run again.
In the picture above you’ll notice a screen hooked up to a board. During the test, the screen looks as pictured below:
So what does this screen mean?
Drive 1 (and Drive 2): These numbers are the variances between test 1, and test 2 on each individual drive system. This means that the difference between the first time Drive 1 ran its test and the second time it ran its test, there was a difference of 7 (or 9) on the Scroll wheel.
The Variance is the difference between the highest test and the lowest test on these two systems. Think of the example below:
Test 1 Test 2
D1: 5000 5007
D2: 4928 4937
The difference between the highest value (5007) and the lowest value (4928) gives the variance of 79.
Note: These numbers were made up for the purpose of a simplified example.
The final reading on the screen is the “bin” number – in this case, bin 1. These bins are sorted by the total amount of steps driven, and organized by different ranges. Each Palette will only have drives from a single bin, which allows us to reduce total variance across all drives.
QC and Calibration Automation
Quality Control (QC) and calibration was one of our biggest production bottlenecks. To give you an idea of the improvements we’d like to talk you through where we started, “Time 0” so to speak.
There are six main steps in the Palette’s QC and calibration process:
1) Steps per count (as mentioned above)
2) Opto to Cutter
3) Splice Distance
4) Splice Test
5) Printer Calibration
6) Print Test
1) Steps per count (S/C): This process involves running a specific number of steps on every ingoing drive, and reading the amount of Scroll Wheel ticks that these steps cause. S/C allows us to calibrate a specific Palette to a specific Scroll Wheel.
8 Tests would be run (2 per drive), 8 numbers recorded, then manually recorded on a piece of paper, and into an excel sheet. This excel sheet would then output a number for steps per count, say, 3.14, which would be manually entered into our stand-alone software during the Palette profile step.
2) Opto to Cutter: Opto to cutter measures the number of steps required for each ingoing drive to drive filament from the cutter module, to the opto sensor.
This process will then output a number that is an average of the four ingoing drives. This number is then recorded on a piece of paper, manually input into an excel sheet, and manually entered into the stand-alone software during the Palette profile step.
3) Splice Distance: This test figures out the number of steps from the splicer’s limit switch, to the optimal splice location. We run a series of tests, each test increases the steps forward by 1, until the splicer arm hits the Teflon and bounces back. Once this happens, we take the number on the screen, write it on a piece of paper, manually input it into an excel sheet, and then manually enter it into the stand-alone software during the Palette profile step.
Once these three tests are complete, we have to slice an .stl to get an .msf file for the next portion, the splice test.
Before we move to the printing stage, we run splice tests to ensure that each Palette is making high quality splices. In order to do so, we had to manually slice an object, pull the .msf file onto the Palette, and setup the Palette in Demo Mode. Then, a number of splices (30+) would be run, and their quality examined.
Once that occurred, we would run a 25 minute calibration print. This calibration print pairs the Palette/Scroll Wheel combo to the test printer. Two numbers are given at the end of this test, recorded on a piece of paper, and an excel sheet, then manually input into the stand-alone software during the print profile step. The standalone then creates files for the final 4 colour test print. The print must be completed successfully for a Palette to be ready to ship.
All in all, this process took over an hour of hands on time per Palette (not including the two hour print test). The setup process for each of the tests, the multiple data entries, the manual nature of the data entries, the profile creations, the file creations and organization, the splice setup, and the print setup was an incredibly daunting project – for every Palette.
….and that’s why we have technology. By taking our first step towards an automated system we were able to cut this time down by half (~30 minutes removed).
How did we do that? Through this simple form, one testing change, and a LOT of math in the background.
As you can probably see at the top, this is our “MSF Writer.” It does a lot of different things:
1) Enters all data into excel sheets
2) Creates Palette profile
3) Creates Printer profile
4) Calibrates Palette to printer
5) Creates demo (splice test) .msf file
6) Creates test print .msf,.gcode
Basically it does everything but run the physical tests, and setup the printer for us.
But…… how can a sheet calibrate a Palette to a printer?
Short answer… it can’t, there’s one last hack we want to talk about before we close off this update.
One of the biggest time requirements used to be the printer to Palette calibration. This calibration consists of running a single colour print, recording the scroll wheel value, and inputting this info into our application (along with the print’s .gcode) to figure out the extrusion constant. We used to do this on every Palette, but we knew that this was a temporary measure.
So, what we did was calibrate a master Scroll Wheel to all test printers. This master wheel is then used to calibrate individual Palettes, based on what printer is selected in the MSF Writer above.
Basically, we did multiple test prints with the master Scroll Wheel, recorded those numbers, did some stuff in the backend, and now by comparing the test scroll wheel to the master Scroll Wheel we can calibrate the Palette to the Printer without having to run a print.
So that’s what’s been going on here the past two weeks. We have a lot of Palettes waiting to be QC’d and calibrated. We’ve been really busy, and we’re making some really big improvements to the systems we have in place here. We're a bit behind the deadlines we laid out in our last post - we will have more information as per the delivery timelines in the next two weeks (after the results of the new Drive QC is made).
We’re getting closer every day, and things are really starting to pick up here – we really appreciate everyone’s continued support.
One last thing! We had so much fun meeting so many of you at the Maker Faire in San Francisco this weekend! You guys had incredible suggestions and feedback that we will plan to work into future developments.
We only do one or two of these events every year – the SF MakerFaire is always an exciting time. Thanks again for coming out!
Until next time….
-Mosaic.