开发者

Threads issues in .net

开发者 https://www.devze.com 2022-12-18 23:11 出处:网络
I have tried to implement a multithreaded crawler and it seems to be working in fetching a list or urls concurrently without any issues.

I have tried to implement a multithreaded crawler and it seems to be working in fetching a list or urls concurrently without any issues. I tested each step and had the program write all html pulled to a text file. Now the rest of the program intends to take each html stored as a string and parse it for a list of urls from that page and then write this list to a database. This is where the errors start: First I have locked out the parsing process since it first caused errors by returning empty lists with the error ' property evaluation failed' Now I have lists being returned but I cannot write this to a database.

My question is, do I need to lock out everything and why? Can I not allow all threads to parse at the same time and each write to an arraylist? Will this all hinder performance?

Here is a sample of some of my code; first the call to go and parse a url:

If Not String.IsNullOrEm开发者_StackOverflow社区pty(html) Then
            'get all links first

            links = parser.GetLinks(fromUrl, html)

then to write to a database:

For Each link As String In links


          recordsAffected = _
                    Links_DBObj.insert_feedurls_link(link, feedlink, execError, connObj_Generic, commObj_Generic)


Instead of using an ArrayList, I would use a Synchronized Queue. Each reading thread can Enqueue while each writing thread can Dequeue.

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号