login about faq

and it doesn't help to issue spider.put_AvoidHttps(False) before crawling

asked Oct 19 '12 at 06:01

oberon62's gravatar image

oberon62
1111


I did not find a problem.

Here's my simple C++ test program:

void spiderTest(void)
    {
    CkSpider spider;

const char *url = "http://www.chilkatsoft.com/crawlStart.html";
const char *domain = "www.chilkatsoft.com";

spider.Initialize(domain);
spider.AddUnspidered(url);
spider.put_CacheDir("c:/aaworkarea/spiderCache");

//  Begin crawling the site by calling CrawlNext repeatedly.
int i,total;
total=0;
for (i = 0; i < 10; i++)
{
    bool success;
    success = spider.CrawlNext();
    if (success == true)
    {
        total++;
        if(spider.get_LastFromCache())
    {
    printf("Downloaded from cache: %s\n",spider.lastUrl());
    }
        else
    {
    printf("Downloaded from Internet: %s\n",spider.lastUrl());

    spider.SleepMs(1000);
    }
    }
    else
    {
    if (spider.get_NumUnspidered() == 0)
    {
    printf("No more URLs to spider\n");
    }
    else
    {
    printf("%s\n",spider.lastErrorText());
    }
    break;

    }

}
}
link

answered Oct 19 '12 at 18:44

chilkat's gravatar image

chilkat ♦♦
11.8k316358421

I have the same problem, I cant crawl the following webpage: https://naxom.se/

what can be the problem? https or the link structure?

link

answered Mar 14 '13 at 19:15

syst3m's gravatar image

syst3m
1111

@system, hi can u provide the code example for your spider ~ i am unable to get mine to execute although there is no error. it just stop after i try run the .exe file.

(Mar 17 '13 at 21:47) GaGoKoYa
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or __italic__
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×13
×13

Asked: Oct 19 '12 at 06:01

Seen: 1,495 times

Last updated: Mar 17 '13 at 21:47

powered by OSQA