I work with a number of process automation systems (DCSs, PLCs). These systems have configuration export files that are not very easy to query due their contetn structure. For one system whose files are especially tedious I created a program to convert the file to an xml structure in VBA - just to see if it could be done without too much complexity. The VBA version is really slow - 1GB file takes 45 minutes.

I woudl liek to rewrite in C# and do some smarter things 0 like buffering the input and ouput streams more and multithreading. I have never written anything multithreaded and wanted to share my idea for how to design the application to get some feedback:

I see four threads ( I intend to use methods that rely on the threadpool):
Read Data from file
Parse to initial xml format
Insert default values that are omitted from export file using XLT
Write data to file

it will take multiple reads to get enough information through the parsing to be able to send it to the transforms. The transfor will be more than enough to send directly to the write.

I am not sure how I keep the threads synchronized to process the right data in the right order.

Thoughts?