Search Results

Search found 24376 results on 976 pages for 'site crawler'.

Page 2/976 | < Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >

• Site to Site VPN problem, connection succesful data only oneway?

  - by Charles

  To start things off, I'm not the actual Administrator for the VPN Server, but he is also at a loss so I thought I'd ask it here. I know it's a Cisco ASA Firewall/VPN. I have a router that connects to the Cisco VPN server, it does so succesfully. I can ping everything within the remote network and from the remote network into my own. I've been able to SSH into a remote server over VPN as well, it all seems to work; until there's some more data returned. A quick example would be an internal webserver. The default homepage simply redirects, so only sends back HTTP headers with a "Location:". I receive this on my computer, but when I request the actual page then (which isn't that big) I don't get a response at all - it just stalls. And it does this for other services as well, for example SSH. I can do a couple of things while connected, but if there's more than xx output it seems to do nothing. The connection remains active throughout all of this. Has anyone ever experienced anything like this before / know what the problem might be? Another user who has a site-to-site connection with this VPN using the -exact same setup- has no problems, the only difference is that I have around 200ms ping to the VPN server/network because of a very long distance (other continent).

  Read the article

• iFrame ads from site to site

  - by user28327

  How I can make the iFrame ads from site A to site B not like this: <iframe src="http://www.site.com"></iframe> Example: site A <script type="text/javascript"><!-- google_ad_client = "ca-pub-3434343507"; /* site */ google_ad_slot = "343435270"; google_ad_width = 728; google_ad_height = 90; //--> </script> <script type="text/javascript" src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script> site B --iFrame-- <script type="text/javascript"><!-- google_ad_client = "ca-pub-3434343507"; /* site */ google_ad_slot = "343435270"; google_ad_width = 728; google_ad_height = 90; //--> </script> <script type="text/javascript" src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script> --iFrame--

  Read the article

• Sharepoint 2010 moving site collection to different database error

  - by Brandon Ulasiewicz

  I am trying to move a site collection from one content database to another content database. First I used the following PowerShell command: New-SPContentDatabase -Name New_DB -WebApplication http://portal/ I confirmed that this did in fact create the DB in the SQL Server. I then used the following command: Move-SPSite http://portal/sites/hr -DestinationDatabase New_DB This generates an error stating that the "Operation is not valid due to the current state of the object" Can anyone help point me in the right direction with this? Thanks

  Read the article

• Do I create new site or add to existing site?

  - by nitbuntu

  Hi, Suppose, as an example, I have a website with the address, www.cool-gifts.com and I'm getting regular sales and its a worthwhile site, but no great fireworks. After research I find that there is a great market for '2nd hand stuff' and I'd like to serve that market. Would it be best to add '2nd hand stuff' as an additional category of gifts in my existing site....or, since the 2nd hand stuff is a market in itself, would I be better off investing time and energy bringing up a whole new site (www.used-stuff.com)? If I had employees and financial resources, it probably would be a no-brainer...start a new site. But, what if you are a small guy, with limited resources? So...new site....or add to existing site?

  Read the article

• Increase traffic to a site through a site on subdomain [closed]

  - by user1716672

  Possible Duplicate: Subdomain versus subdirectory We have two sites, one is mainly a portfolio site (built with Yii framework) and the other is a digital shop (built with open cart) where we sell plugins and themes. The url's look like www.mydomian.com and www.store.mydomain.com. But of these sites are in the same server. We use google analytics tools and have no problem getting traffic to our store. But we have very little to our portfolio site and we want to increase our Google ranking for this site. Assuming increased traffic to our site will increase our google ranking, we were thinking to use URl masking so the link will be www.mydomain.com/shop and this will load www.store.mydomain.com. Will this count as hits for our portfolio site? Because the .htaccess rules will ensure the subdomain is served. So I dont know if these hits will count on our store or our portfolio site...

  Read the article

• can a crawler be written entirely in javascript?

  - by goss

  I was wondering - can a crawler be written entirely in javascript? That way, the crawler is only called when a user needs the information and everything is run from the individual user's computer

  Read the article

• How to write a crawler?

  - by Jason

  Hi All, I have had thoughts of trying to write a simple crawler that might crawl and produce a list of its findings for our NPO's websites and content. Does anybody have any thoughts on how to do this? Where do you point the crawler to get started? How does it send back its findings and still keep crawling? How does it know what it finds, etc,etc. Thanks! -Jason

  Read the article

• Cisco ASA - VPN and Hairpinning....

  - by Nordberg

  Hi, We have 2 sites that will be linked by a IPSEC VPN between 2 Cisco ASAs: Site 1 8Mb ADSL Connection Cisco ASA 505 Site 2 2Mb SDSL Connection Cisco ASA 505 Basically, both sites need access to a service at the end of another IPSEC VPN, Site 3, which I plan to terminate at Site 2. This is due to the way the service is sold - it's billed per gateway. So if both Site 1 and Site 2 had their own VPN connection to Site 3, it would cost us twice as much... Anyway, my idea is to have all traffic from Site 1 destined for Site 3 to go via the VPN between Site 1 and Site 2. The end result being all traffic that hits Site 3 has come via Site 2. I understand this is known as hairpinning but I'm struggling to find a great deal of information on how this is setup. So, firstly, can anyone confirm that what I'm trying to achieve is possible and, secondly, can anyone point me in the direction of an example of such a configuration? Many Thanks.

  Read the article

• Crawler do not create custom crawled properties

  - by user173739

  These days i have faced with very strange problem. I have development environment with MOSS 2007 SP 2 and WS 2008, i have search configured and everything works great. I have started to configuring staging environment (MOSS 2007 SP2 with June CU) and create new farm and new SSP. I have deployed my changes with package (wsp) and manually create site collections, sub webs, pages and so on. When fill crawl finishes, i see in Crawl log that all my pages have been successfully crawled and when i use some test tools to query search, my pages have been found. In crawl log there is few errors like http://mysite/sites/de/pages "The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly..", but all pages in this Page library were indexed. The problem is that i use custom managed properties (mapped to custom crawled properties) in search queries, but crawler didn't create crawled properties for all my new site columns. For example for site column IsAccent the crawler didn't create cralwed property ows_isAccesnt. I'm sure that i have created pages for specific content type and all my crawl categories have checked "Automatically discover new properties when a crawl takes place ". In site settings - Searchable columns i haven't got any column selected as Nocrowl. I tried to export my managed and crawled properties from dev environment to stage evironment but all my managed properties were empty, after that i recreated SSP...the result was the same... I checked specific page with tools like Sharepoint Manager 2007 and U2U Caml Query Builder 2007 that content type is correct, and i can see values of my custom site collumns.... Using U2U Caml Query Builder 2007 agains some Page library in Result tab i can see ows_IsAccent (my site collumn is IsAccent) and others site columns, but i can't find them in Crawled properties. Any idias?

  Read the article

• Remove subdomain from Google Crawler

  - by Walter White

  Hi all, I recently removed a sub-domain from my domain so I just have 1 website to manage. However, if I do a google search, my old domain is still there, I removed the sub-domain well over a week ago and if you try to access the domain directly, you will get an error saying the website can not be found (the records have been deleted). What is the easiest way to remove that sub-domain from google searches since they no longer exist? Shouldn't google see that the domain no longer exists and delete those entries? Walter

  Read the article

• Website crawler/spider to get site map

  - by ack__

  I need to retrieve a whole website map, in a format like : http://example.org/ http://example.org/product/ http://example.org/service/ http://example.org/about/ http://example.org/product/viewproduct/ I need it to be linked-based (no file or dir brute-force), like : parse homepage - retrieve all links - explore them - retrieve links, ... And I also need the ability to detect if a page is a "template" to not retrieve all of the "child-pages". For example if the following links are found : http://example.org/product/viewproduct?id=1 http://example.org/product/viewproduct?id=2 http://example.org/product/viewproduct?id=3 I need to get only once the http://example.org/product/viewproduct I've looked into HTTtracks, wget (with spider-option), but nothing conclusive so far. The soft/tool should be downloadable, and I prefer if it runs on Linux. It can be written in any language. Thanks

  Read the article

• HTTP crawler in Erlang

  - by ctp

  I'm coding on a simple HTTP crawler but I have an issue running the code at the bottom. I'm requesting 50 URLs and get the content of 20+ back. I've generated few files with 150kB size each to test the crawler. So I think the 20+ responses are limited by the bandwidth? BUT: how to tell the Erlang snippet not to quit until the last file is not fetched? The test data server is online, so plz try the code out and any hints are welcome :) -module(crawler). -define(BASE_URL, "http://46.4.117.69/"). -export([start/0, send_reqs/0, do_send_req/1]). start() -> ibrowse:start(), proc_lib:spawn(?MODULE, send_reqs, []). to_url(Id) -> ?BASE_URL ++ integer_to_list(Id). fetch_ids() -> lists:seq(1, 50). send_reqs() -> spawn_workers(fetch_ids()). spawn_workers(Ids) -> lists:foreach(fun do_spawn/1, Ids). do_spawn(Id) -> proc_lib:spawn_link(?MODULE, do_send_req, [Id]). do_send_req(Id) -> io:format("Requesting ID ~p ... ~n", [Id]), Result = (catch ibrowse:send_req(to_url(Id), [], get, [], [], 10000)), case Result of {ok, Status, _H, B} -> io:format("OK -- ID: ~2..0w -- Status: ~p -- Content length: ~p~n", [Id, Status, length(B)]); Err -> io:format("ERROR -- ID: ~p -- Error: ~p~n", [Id, Err]) end. That's the output: Requesting ID 1 ... Requesting ID 2 ... Requesting ID 3 ... Requesting ID 4 ... Requesting ID 5 ... Requesting ID 6 ... Requesting ID 7 ... Requesting ID 8 ... Requesting ID 9 ... Requesting ID 10 ... Requesting ID 11 ... Requesting ID 12 ... Requesting ID 13 ... Requesting ID 14 ... Requesting ID 15 ... Requesting ID 16 ... Requesting ID 17 ... Requesting ID 18 ... Requesting ID 19 ... Requesting ID 20 ... Requesting ID 21 ... Requesting ID 22 ... Requesting ID 23 ... Requesting ID 24 ... Requesting ID 25 ... Requesting ID 26 ... Requesting ID 27 ... Requesting ID 28 ... Requesting ID 29 ... Requesting ID 30 ... Requesting ID 31 ... Requesting ID 32 ... Requesting ID 33 ... Requesting ID 34 ... Requesting ID 35 ... Requesting ID 36 ... Requesting ID 37 ... Requesting ID 38 ... Requesting ID 39 ... Requesting ID 40 ... Requesting ID 41 ... Requesting ID 42 ... Requesting ID 43 ... Requesting ID 44 ... Requesting ID 45 ... Requesting ID 46 ... Requesting ID 47 ... Requesting ID 48 ... Requesting ID 49 ... Requesting ID 50 ... OK -- ID: 49 -- Status: "200" -- Content length: 150000 OK -- ID: 47 -- Status: "200" -- Content length: 150000 OK -- ID: 50 -- Status: "200" -- Content length: 150000 OK -- ID: 17 -- Status: "200" -- Content length: 150000 OK -- ID: 48 -- Status: "200" -- Content length: 150000 OK -- ID: 45 -- Status: "200" -- Content length: 150000 OK -- ID: 46 -- Status: "200" -- Content length: 150000 OK -- ID: 10 -- Status: "200" -- Content length: 150000 OK -- ID: 09 -- Status: "200" -- Content length: 150000 OK -- ID: 19 -- Status: "200" -- Content length: 150000 OK -- ID: 13 -- Status: "200" -- Content length: 150000 OK -- ID: 21 -- Status: "200" -- Content length: 150000 OK -- ID: 16 -- Status: "200" -- Content length: 150000 OK -- ID: 27 -- Status: "200" -- Content length: 150000 OK -- ID: 03 -- Status: "200" -- Content length: 150000 OK -- ID: 23 -- Status: "200" -- Content length: 150000 OK -- ID: 29 -- Status: "200" -- Content length: 150000 OK -- ID: 14 -- Status: "200" -- Content length: 150000 OK -- ID: 18 -- Status: "200" -- Content length: 150000 OK -- ID: 01 -- Status: "200" -- Content length: 150000 OK -- ID: 30 -- Status: "200" -- Content length: 150000 OK -- ID: 40 -- Status: "200" -- Content length: 150000 OK -- ID: 05 -- Status: "200" -- Content length: 150000 Update: thanks stemm for the hint with the wait_workers. I've combined your and mine code but same behaviour :( -module(crawler). -define(BASE_URL, "http://46.4.117.69/"). -export([start/0, send_reqs/0, do_send_req/2]). start() -> ibrowse:start(), proc_lib:spawn(?MODULE, send_reqs, []). to_url(Id) -> ?BASE_URL ++ integer_to_list(Id). fetch_ids() -> lists:seq(1, 50). send_reqs() -> spawn_workers(fetch_ids()). spawn_workers(Ids) -> %% collect reference to each worker Refs = [ do_spawn(Id) || Id <- Ids ], %% wait for response from each worker wait_workers(Refs). wait_workers(Refs) -> lists:foreach(fun receive_by_ref/1, Refs). receive_by_ref(Ref) -> %% receive message only from worker with specific reference receive {Ref, done} -> done end. do_spawn(Id) -> Ref = make_ref(), proc_lib:spawn_link(?MODULE, do_send_req, [Id, {self(), Ref}]), Ref. do_send_req(Id, {Pid, Ref}) -> io:format("Requesting ID ~p ... ~n", [Id]), Result = (catch ibrowse:send_req(to_url(Id), [], get, [], [], 10000)), case Result of {ok, Status, _H, B} -> io:format("OK -- ID: ~2..0w -- Status: ~p -- Content length: ~p~n", [Id, Status, length(B)]), %% send message that work is done Pid ! {Ref, done}; Err -> io:format("ERROR -- ID: ~p -- Error: ~p~n", [Id, Err]), %% repeat request if there was error while fetching a page, do_send_req(Id, {Pid, Ref}) %% or - if you don't want to repeat request, put there: %% Pid ! {Ref, done} end. Running the crawler forks fine for a handful of files, but then the code even doesnt fetch the entire files (file size each 150000 bytes) - he crawler fetches some files partially, see the following web server log :( 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /10 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /1 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /3 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /8 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /39 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /7 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /6 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /2 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /5 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /50 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /9 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /44 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /38 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /47 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /49 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /43 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /37 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /46 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /48 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /36 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /42 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /41 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /45 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /17 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /35 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /16 HTTP/1.1" 200 150000 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /15 HTTP/1.1" 200 17020 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /21 HTTP/1.1" 200 120360 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /40 HTTP/1.1" 200 117600 "-" "-" 82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /34 HTTP/1.1" 200 60660 "-" "-" Any hints are welcome. I have no clue what's going wrong there :(

  Read the article

• Google Site Search -- How to use as API?

  - by John Isaacks

  I am trying to get an API that I can use to do searches on my own site. Google has something called site search and something called custom search. What is the difference? I make a new site search, then it is listed on a page with "custom search" in the heading. This is really confusing. I just want an API that I can use to search my site. I would prefer json to xml as the results. And if this service is offered by someone other than Google, that is fine too. The ones that I create at Google want me to embed a premade search box into my site. I do not want that, I want an API that I can call from PHP or JS. How can I get this?

  Read the article

• how to get new site indexed by alexa

  - by JohnMerlino

  When I search my site on alexa, it says "Alexa Traffic Rank: No Data". So I google the issue. I come across this page: http://www.loudable.com/my-website-data-not-showing-in-alexa-get-your-website-crawled-by-alexasolution.html It says to get site indexed, click Crawl my site on the webmasters page. However there is no longer a link that says "Crawl my site". So as of now, does anyone know how to get site indexed by alexa so that my traffic rank will display on alexa index?

  Read the article

• My site meta description changes and pointing to some casino site [on hold]

  - by prateekrc3

  meta description contains link to "http://infoexgraphics.com/spices/?p=no-deposit-casino-bonus" some casino or gaming site I have no idea where this come from. here is the screen shot http://tinypic.com/r/2yzdi7s/8 I have already search code but found none. When i tried to use search operator with keyword "site:jobdaddy.in casino" i found that all of my pages suffered from this behavior? Is my site hacked ?? Pls help My site is jobdaddy.in Thanks :)

  Read the article

• Non-public site for testing on shared-hosting site

  - by ptpaterson

  Is it possible to as a developer using a shared hosting site such as bluehost, hostgator, and the like, to view your site without making it public. Or do the files you upload always go live immediately? Is the best way to test a site (if using shared hosting) to just set up some apache/mysql/php service on my machine? I am considering putting together a site with shared hosting, and trying to see what all my options are. Thanks.

  Read the article

• If an visitors IP address contains "google" or a similar keyword, does this mean they were a crawler?

  - by Roscoe

  Hi, I have a huge list of IP addresses recorded from various visitors to a website. A huge amount of the visitors, in some months over 70%, came from IP addresses that contained keywords such as google, yahoo, bot, crawler, etc. Does this mean that those users were infact search engine crawlers? If so, why are their so many crawlers in my visitor records in comparison to genuine human visitors? (and if not what's the explanation?) Thanks in advance.

  Read the article

• Site Studio Mobile Example - WCM Reuse

  - by john.brunswick

  Mobile internet usage is growing by leaps and bounds and it is theorized that in the not-to-distant future it will eclipse traditional access via desktop browsers. Mary Meeker, a managing director at Morgan Stanley and head of their global technology research team, recently predicted that mobile usage will eclipse desktop usage within the next 5 years in an Events@Google series presentation. In order for organizations to reach their prospects, customers and business partners, they will need to make their content readily available on mobile devices. A few years ago it was fairly challenging to provide a special, separate, site to cater to mobile users using technologies like WML (Wireless Markup Language). Modern mobile browsers have rendered the need for this as irrelevant and now the focus has moved toward providing a browsing experience that works well on small screen sizes and is highly performant. What does all of this mean for Oracle UCM? Taking site content from an existing Site Studio site and targeting it for consumption for mobile devices is a very straightforward process that is aided by a number of native capabilities in the product. The example highlighted in this post takes advantage of dynamic conversion capabilities in Oracle UCM to enable site content to be created and updated via MS Office documents. These documents are then converted to a simple, clean HTML format for consumption in the desktop and mobile browsing experiences. To help better understand how this is possible the example below shows a fictional .COM and its mobile site counterpart that both leverage the same underlying content. The scenario is not complete or production ready, but highlights that a mobile experience may be best delivered by omitting portions of a site that would be present within the version served to desktop clients. If you have browsed CNet (news.com) on a mobile device it becomes quickly apparent that they are serving an optimized version for your mobile device. An iPhone style version can be accessed at http://iphone.cnet.com/. In order to do that they leveraged some work done for the iPhone iUi project developed by Joe Hewitt that provides mobile browsers an experience that is similar to what users may find in a native iPhone application. For our example parts of this framework are used (the CSS) and this approach provides a page that will degrade nicely over a wide range of mobile browsers, since it is comprised of lightweight HTML markup and CSS. The iPhone iUi framework also provides some nice JavaScript to enable animated transitions between pages, but for the widest range of mobile browser compatibility we will only incorporate the CSS and HTML DIV / UL based page markup in our example.

  Read the article

• How can I transfer article content from old Joomla 1.5 site to new 2.5 site

  - by PaulHurleyuk

  I have an existing Joomla 1.5 site and am intending to wipe it and install a brand new 2.5 site. I will pick new plugins, template etc but would like to transfer the basic text / images of the articles on the 1.5 site to the new site. I am less concerned with categories and tags of those old articles, as they'll probably go in an 'old' category. I have several file and database backups of the 1.5 site. Has anyone done anything similar ? Are the two article db schemas similar enough to just transfer the data ?

  Read the article

• Building an automatic web crawler

  - by Sakin

  I am building a web application crawler that's meant not only to find all the links or pages in a web application, but also perform all the allowed actions in the app (such as pushing buttons, filling forms, notice changes in the DOM even if they did not trigger a request etc.) Basically, this is a kind of "browser simulator". I find WebKit a good option to implement my crawler, since it has all the needed technology (Javascript engine, parsers, DOM manipulation, etc.) but it seems kind of an overkill being a fully featured browser. Is there any toolkit you know that can provide the above functionality?

  Read the article

• What is a good Java crawler library?

  - by DrDee

  Hi, I am about to develop a crawler in Java but don't feel like reinventing the wheel. A quick Google search gives a whole bunch of Java libraries to build a web crawler. Besides that Nutch is of course a very robust package but seems a bit too advanced for my needs. I only need to crawl a handful websites a week containing a couple of 1000 pages each. Which open source Java library would you recommend considering: speed multithreading (or even distributed) extending it with new functionality active maintained and documentation?

  Read the article

• What is a good Java web crawler library?

  - by DrDee

  Hi, I am about to develop a crawler in Java but don't feel like reinventing the wheel. A quick Google search gives a whole bunch of Java libraries to build a web crawler. Besides that Nutch is of course a very robust package but seems a bit too advanced for my needs. I only need to crawl a handful websites a week containing a couple of 1000 pages each. Which open source Java library would you recommend considering: speed multithreading (or even distributed) extending it with new functionality active maintained and documentation?

  Read the article

• htacces rewrite condition old site to new site with querystring

  - by Brandon Braner

  I am not even going to pretend to fully understand how htaccess rewrite conidtions work. Ive been working on this for a while searching and searching. I have an old wordpress site www.old-site.com and a new site www.site.com wordpress uses query strings page_id=# to redirect to pages on the old site page_id=2 went to a specific page but on the new site it goes the the home page i need old-site/?page_id=2 to go to site.com/our-company here is what i am trying RewriteCond %{HTTP_HOST} ^(www.)?old-site.com$ [NC] RewriteCond %{QUERY_STRING} ^page_id=2$ RewriteRule ^(.*)$ http://www.site.com/our-company/ [R=301,L] if i take out the rewrite conditio for query string it redirects all traffic from old-site.com to the our company page on the new site. where am i going wrong? i have about 15 redirects i need to do this way. thanks in advance

  Read the article

• htaccess rewrite condition old site to new site with querystring

  - by Brandon Braner

  I am not even going to pretend to fully understand how htaccess rewrite conditions work. I've been working on this for a while searching and searching. I have an old Wordpress site www.old-site.com and a new site www.site.com. Wordpress uses query strings page_id=# to redirect to pages. On the old site page_id=2 went to a specific page but on the new site it goes the the home page. I need old-site/?page_id=2 to go to site.com/our-company Here is what I am trying RewriteCond %{HTTP_HOST} ^(www\.)?old-site.com$ [NC] RewriteCond %{QUERY_STRING} ^page_id=2$ RewriteRule ^(.*)$ http://www.site.com/our-company/ [R=301,L] If I take out the rewrite condition for query string it redirects all traffic from old-site.com to the company page on the new site. Where am I going wrong? I have about 15 redirects I need to do this way.

  Read the article

• Cracking Websites with Cross Site Scripting

  - by Akemi Iwaya

  You may have heard the term ‘cross site scripting’ before, but what exactly is ‘it’ and why is it dangerous for a website? YouTube channel Computerphile presents a nice primer on the ‘how and why’ of cross site scripting and the dangers it presents in their latest video. Cracking Websites with Cross Site Scripting – Computerphile [YouTube]     

  Read the article

< Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >