A few things to know before hand
Steps:
1. Use http request to get the file and store it temporarily.(Prefer .txt/.csv in this case :) )
2. Initiate a server thread to process the file.
3. The thread read line by line and separate the data by the delimiters.
4. Then process data and validate it(Regular Expressions).
5. Use batch executes to store data in database. Do not use single Insert/updates as the locks will open too many statements and cause a system crash.
6. At end of processing execute the batch, clear memory.
Note more system memory required for more users and use user limits(waiting locks to manage memory) and updated garbage collection
- User thread bulk data processing is slow so use separate threads than usr
- Oracle uses locking to for insert and updates. Using batch statements provide partial solutions
Steps:
1. Use http request to get the file and store it temporarily.(Prefer .txt/.csv in this case :) )
2. Initiate a server thread to process the file.
3. The thread read line by line and separate the data by the delimiters.
4. Then process data and validate it(Regular Expressions).
5. Use batch executes to store data in database. Do not use single Insert/updates as the locks will open too many statements and cause a system crash.
6. At end of processing execute the batch, clear memory.
Note more system memory required for more users and use user limits(waiting locks to manage memory) and updated garbage collection
No comments:
Post a Comment