GiNi 2 U d Z ddlZddlmc mZ ddlZddlZddl Z ddl
Z
ddl m Z ddl
mZ ddlmZmZ ddlmZmZmZ ddlZe j. j1 d e e e j6 j6 dZ ee e d< ddl Z e du Z ejB jE e d Z#d
Z$d Z%d Z& d!d
edede'defdZ(dedede)e fdZ* G d d Z+ G d d Z, G d d Z- G d d Z. G d d Z/ G d d Z0y# e $ r Y w xY w)"u TDD RED 단계: test_insurance_spider.py
Scrapling Spider ABC를 상속한 InsuranceSpider 클래스와
ResponseHistory 유틸리티에 대한 테스트 스위트.
(insurance_spider.py는 아직 구현되지 않음 - TDD RED 단계)
주의:
- 모든 테스트는 로컬에서 수행 (mock 사용). 외부 네트워크 호출 없음.
- 실제 크롤링 시 반드시 대상 사이트의 robots.txt를 확인하고 준수해야 합니다.
- 합법적 공개 데이터(보험사 공시 페이지 등)만을 대상으로 합니다.
N)Path)
ModuleType)AnyOptional) AsyncMock MagicMockpatch_insurance_spideruH insurance_spider.py 미구현 (TDD RED 단계 - 토르가 구현 예정))reasonu
화재보험
50000
자동차보험
100000
생명보험
30000
다음
u
| 보험명 | 보험료 | 가입기간 |
|---|
| 화재보험 | 50000 | 1년 |
| 자동차보험 | 100000 | 1년 |
z
htmlurlstatusreturnc D ddl m} || |d }||_ i |_ |S )u Selector를 Response 대용으로 사용 (Response는 Selector 상속).
추가 속성(status, meta)만 동적으로 붙인다.
r )SelectorF)r
adaptive)scrapling.parserr r meta)r r
r r sels :/home/jay/workspace/scripts/tests/test_insurance_spider.pymake_mock_responser X s( *
4S5
1CCJCH J spiderresponsec p K g }| j | 2 3 d{ }|j | 7 6 |S w)u9 parse() async generator의 결과를 리스트로 수집.N)parseappend)r r resultsitems r collect_parse_resultsr i s= G ll8, dt , Ns 6313636c e Zd ZdZed d Zed d Zed d Zed d Zed d Z ed d Z
ed d Z ed d
Z y) TestResponseHistoryu$ ResponseHistory 클래스 테스트.Nc ` d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j }|j } | }g }||k( }|st j d|fd ||f d
t j v st j
| rt j | nd
t j | t j | t j | d z } d d
| iz }
t t j |
dx}x}x}}y)u: 기본 생성 시 히스토리가 비어 있어야 한다.Nis notz%(py0)s is not %(py3)sr
py0py3assert %(py5)spy5==zJ%(py4)s
{%(py4)s = %(py2)s
{%(py2)s = %(py0)s.get_history
}()
} == %(py7)shistoryr( py2py4py7assert %(py9)spy9) r
@pytest_ar_call_reprcompare @py_builtinslocals_should_repr_global_name _safereprAssertionError_format_explanationResponseHistory get_history) self @py_assert2 @py_assert1 @py_format4 @py_format6r/ @py_assert3 @py_assert6 @py_assert5 @py_format8 @py_format10s r test_default_initz%TestResponseHistory.test_default_inity s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,#335""*"$**$****$******w***w***"***$**********r c d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j }|j dd |j }t | }d
}||k( }|st j d |fd ||f d
t j v st j
t rt j t nd
dt j v st j
| rt j | ndt j | t j | dz } dd| iz }
t t j |
dx}x}}|d d } d} | | k( }|slt j d |fd| | f t j | t j | dz }
dd|
iz } t t j | dx} x}} |d d } d } | | k( }|slt j d |fd| | f t j | t j | dz }
dd|
iz } t t j | dx} x}} y)u< record() 후 get_history()에 항목이 나타나야 한다.Nr$ r& r
r' r* r+ https://example.com r, z0%(py3)s
{%(py3)s = %(py0)s(%(py1)s)
} == %(py6)slenresultr( py1r) py6assert %(py8)spy8r r
z%(py1)s == %(py4)srS r2 assert %(py6)srT r
r
r6 r7 r8 r9 r: r; r< r= r> recordr? rP r@ rA rB rC rD r/ rQ rG @py_assert4 @py_format7 @py_format9 @py_assert0rE @py_format5s r test_record_and_get_historyz/TestResponseHistory.test_record_and_get_history s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,#335,c2$$&6{ a {a {a s s 6 6 { a ay 8#88 #88888 #8888 888#88888888ay")c)"c))))"c)))")))c)))))))r c T d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j }|j dd |j d
d |j d d
|j }t | }d}||k( }|st j d|fd||f dt j v st j
t rt j t nddt j v st j
| rt j | ndt j | t j | dz } dd| iz }
t t j |
dx}x}}|d d } d} | | k( }|slt j d|fd| | f t j | t j | dz }
dd|
iz } t t j | dx} x}} |d d } d } | | k( }|slt j d|fd| | f t j | t j | dz }
dd|
iz } t t j | dx} x}} |d d } d
} | | k( }|slt j d|fd| | f t j | t j | dz }
dd|
iz } t t j | dx} x}} y) uM 여러 번 record() 시 모든 항목이 순서대로 보존되어야 한다.Nr$ r& r
r' r* r+ https://example.com/arM https://example.com/b- zhttps://example.com/ci r, rO rP rQ rR rU rV r r
rW rX rY rT rN r rZ r\ s r test_record_multiple_entriesz0TestResponseHistory.test_record_multiple_entries s< )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,#335.4.4.4$$&6{ a {a {a s s 6 6 { a ay :#:: #::::: #:::: :::#::::::::ay")c)"c))))"c)))")))c)))))))ay")c)"c))))"c)))")))c)))))))r c ~ d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j }|j dd |j d
d |j d }t | }d }||k\ }|st j d
|fd||f dt j v st j
t rt j t nddt j v st j
| rt j | ndt j | t j | dz } dd| iz }
t t j |
dx}x}}d |D }t | } | sddt j v st j
t rt j t ndt j | t j | dz } t t j | dx}} y)uE get_chain()은 특정 URL에 연결된 항목만 반환해야 한다.Nr$ r& r
r' r* r+ rd rM re rf rN >=z0%(py3)s
{%(py3)s = %(py0)s(%(py1)s)
} >= %(py6)srP chainrR rU rV c 3 , K | ] }|d dk( yw)r
rd N ).0entrys r z@TestResponseHistory.test_get_chain_single_url.. s Nu5<#::Ns ,assert %(py4)s
{%(py4)s = %(py0)s(%(py2)s)
}anyr( r1 r2 )r
r6 r7 r8 r9 r: r; r< r= r> r[ get_chainrP ru )
r@ rA rB rC rD r/ rn rG r] r^ r_ rE ra s
r test_get_chain_single_urlz-TestResponseHistory.test_get_chain_single_url s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,#335.4.4!!"9:5z Q zQ zQ s s 5 5 z Q NNNsNNNNNNNNNsNNNsNNNNNNNNNNNNNNr c d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j }dd g}|j d
d | |j d
}t | }d
}||k\ } | st j d| fd||f dt j v st j
t rt j t nddt j v st j
| rt j | ndt j | t j | dz }
dd|
iz } t t j | dx}x} }|d } d}
|
| v }|st j d|fd|
| f t j |
dt j v st j
| rt j | nddz }dd|iz }t t j | dx}
}| d }
|
|k( }|st j d|fd |
|f t j |
dt j v st j
| rt j | nddz }dd|iz }t t j | dx}
}y) uL redirects 인자를 통해 리다이렉트 체인이 보존되어야 한다.Nr$ r& r
r' r* r+ zhttps://example.com/oldz https://example.com/redirectzhttps://example.com/newrM ) redirectsrN rk rm rP rn rR rU rV r rz inz%(py1)s in %(py3)srr rS r) r, )z%(py1)s == %(py3)s)
r
r6 r7 r8 r9 r: r; r< r= r> r[ rw rP )r@ rA rB rC rD r/ rz rn rG r] r^ r_ rr r` s r test_redirect_chain_preservedz1TestResponseHistory.test_redirect_chain_preserved s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,#335.0NO %
!!";<5z Q zQ zQ s s 5 5 z Q a#{e####{e###{######e###e#######[!.!Y....!Y...!......Y...Y.......r c
d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j }dd d
}|j d d |
|j }t | }d}||k( } | st j d| fd||f dt j v st j
t rt j t nddt j v st j
| rt j | ndt j | t j | dz }
dd|
iz } t t j | dx}x} }|d } | j }d} || }
|
|k( }|st j d|fd|
|f t j | t j | t j | t j |
dt j v st j
| rt j | nddz }dd|iz }t t j | dx} x}x} x}
}y) u3 headers 인자가 기록에 포함되어야 한다.Nr$ r& r
r' r* r+ z text/htmlvalue)z Content-TypezX-CustomrL rM )headersrN r, rO rP rQ rR rU rV r r )zI%(py7)s
{%(py7)s = %(py3)s
{%(py3)s = %(py1)s.get
}(%(py5)s)
} == %(py9)s)rS r) r+ r3 r5 zassert %(py11)spy11)r
r6 r7 r8 r9 r: r; r< r= r> r[ r? rP get)r@ rA rB rC rD r/ r rQ rG r] r^ r_ r` rF @py_assert8rI @py_format12s r test_record_with_headersz,TestResponseHistory.test_record_with_headers s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,#335#.GD,c7C$$&6{ a {a {a s s 6 6 { a ay2y}}2Y2}Y'2'72222'7222y222}222Y222'2222227222722222222r c | d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j }|j dd |j d
d t j 5 }t | d z }|j t | |j } | }|sd d
t j v st j
| rt j | nd
t j | t j | dz } t t j | dx}}t# j$ |j' d }
t) |
t* }|s ddt j v st j
t( rt j t( nddt j v st j
|
rt j |
nddt j v st j
t* rt j t* ndt j | dz } t t j | d}t- |
}d} || k( } | st j d| fd|| f dt j v st j
t, rt j t, nddt j v st j
|
rt j |
ndt j | t j | dz }
dd |
iz }t t j | dx}x} } |
d d }d}||k( }|slt j d|fd ||f t j | t j | d z } d!d"| iz }
t t j |
dx}x}}ddd y# 1 sw Y yxY w)#u= save()는 히스토리를 JSON 파일로 저장해야 한다.Nr$ r& r
r' r* r+ rL rM zhttps://example.com/page2z history.jsonAassert %(py4)s
{%(py4)s = %(py2)s
{%(py2)s = %(py0)s.exists
}()
}pathrv zutf-8)encoding5assert %(py4)s
{%(py4)s = %(py0)s(%(py1)s, %(py2)s)
}
isinstancedatalistr( rS r1 r2 rh r, rO rP rR rU rV r r
rW rX rY rT )r
r6 r7 r8 r9 r: r; r< r= r> r[ tempfileTemporaryDirectoryr savestrexistsjsonloads read_textr r rP )r@ rA rB rC rD r/ tmpdirr rE ra r rG r] r^ r_ r` s r test_save_to_json_filez*TestResponseHistory.test_save_to_json_file s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,#335,c22C8
(
(
* ;f<.0D LLT #;; ;= = 4 4 ; = ::dnngn >?D dD) )) ) ) ) ) ): ) ) ): ) ) ) ) ) )d ) ) )d ) ) ) ) ) )D ) ) )D ) ) )) ) ) ) ) ) )t9 ! !9> ! ! !9 ! ! ! ! ! !3 ! ! !3 ! ! ! ! ! !t ! ! !t ! ! !9 ! ! ! ! ! ! ! ! ! !75> :%: :>%:: : : :>%: : : :> : : :%: : : : : : : : ; ; ;s
N'R22R;c D d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j }|j dd |j } | }t | }d
}||k( } | s t j d | fd ||f d
t j v st j
t rt j t nd
dt j v st j
| rt j | ndt j | t j | t j | t j | dz }
dd|
iz } t t j | dx}x}x}x} }|j |j } | } g }| |k( }
|
st j d |
fd| |f dt j v st j
| rt j | ndt j | t j | t j | dz }dd|iz }t t j | dx}x} x}
}y)u4 clear() 후 히스토리가 비어 있어야 한다.Nr$ r& r
r' r* r+ rL rM rN r, )zi%(py7)s
{%(py7)s = %(py0)s(%(py5)s
{%(py5)s = %(py3)s
{%(py3)s = %(py1)s.get_history
}()
})
} == %(py10)srP r/ )r( rS r) r+ r3 py10zassert %(py12)spy12r. r0 r4 r5 )r
r6 r7 r8 r9 r: r; r< r= r> r[ r? rP clear)r@ rA rB rC rD r/ r] rF @py_assert9r @py_format11 @py_format13rE rG rH rI s r
test_clearz TestResponseHistory.test_clear s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,#335,c2&&.&(.s().Q.)Q....)Q......s...s......7...7...&...(...)...Q.......
""*"$**$****$******w***w***"***$**********r r N)
__name__
__module__ __qualname____doc___skip_if_missingrJ rb ri rx r r r r rp r r r" r" v s .+ + * * * * O O / / 3 3 ; ; + +r r" c e Zd ZdZed d Zed d Zed d Zed d Zed d Z ed d Z
ed d Z ed d
Z y) TestInsuranceSpiderInitu$ InsuranceSpider 초기화 테스트.Nc Z d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j }|j }d}||k( }|st j d |fd
||f dt j v st j
t rt j t ndt j | t j | t j | d z }d d
|iz } t t j | dx}x}x}}y)u/ name 속성이 'insurance_spider'여야 한다.Nr$ r& r
r' r* r+ insurance_spiderr, )zQ%(py4)s
{%(py4)s = %(py2)s
{%(py2)s = %(py0)s.InsuranceSpider
}.name
} == %(py7)sr0 r4 r5 ) r
r6 r7 r8 r9 r: r; r< r= InsuranceSpidername)
r@ rA rB rC rD rE rF rG rH rI s
r test_name_attributez+TestInsuranceSpiderInit.test_name_attribute s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,, 00K055K9KK59KKKKK59KKKKKKK KKK KKK0KKK5KKK9KKKKKKKKr c L d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j dg }|j }d
}||k( }|st j d |fd ||f d
t j v st j
| rt j | nd
t j | t j | dz }dd|iz }t t j | dx}x}}|j }d}||k( }|st j d |fd||f d
t j v st j
| rt j | nd
t j | t j | dz }dd|iz }t t j | dx}x}}|j }d}||k( }|st j d |fd||f d
t j v st j
| rt j | nd
t j | t j | dz }dd|iz }t t j | dx}x}}y)u\ concurrent_requests=4, concurrent_requests_per_domain=2, download_delay=1.0이어야 한다.Nr$ r& r
r' r* r+ rL
start_urls r, )z;%(py2)s
{%(py2)s = %(py0)s.concurrent_requests
} == %(py5)sr r( r1 r+ assert %(py7)sr3 rh )zF%(py2)s
{%(py2)s = %(py0)s.concurrent_requests_per_domain
} == %(py5)sg ?)z6%(py2)s
{%(py2)s = %(py0)s.download_delay
} == %(py5)s)
r
r6 r7 r8 r9 r: r; r< r= r concurrent_requests concurrent_requests_per_domaindownload_delay r@ rA rB rC rD r r] rE rH s r !test_concurrent_requests_defaultsz9TestInsuranceSpiderInit.test_concurrent_requests_defaults s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,"22?T>UV)).Q.)Q....)Q......v...v...)...Q.......4499499994999999v999v99949999999999$$++$++++$++++++v+++v+++$++++++++++r c d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}dd g}t j |
}|j }||k( }|st j d |fd ||f d
t j v st j
| rt j | nd
t j | dt j v st j
| rt j | nddz }dd|iz } t t j | dx}}y)u= 생성자에서 start_urls를 설정할 수 있어야 한다.Nr$ r& r
r' r* r+ rd re r r, )z2%(py2)s
{%(py2)s = %(py0)s.start_urls
} == %(py4)sr urlsrv rY rT ) r
r6 r7 r8 r9 r: r; r< r= r r )
r@ rA rB rC rD r r rE ra r^ s
r test_start_urls_setz+TestInsuranceSpiderInit.test_start_urls_set s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,')@A"22dC ( D(((( D((((((v(((v((( ((((((D(((D(((((((r c d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}dd h}t j d
g| }|j }||k( }|st j d |fd
||f dt j v st j
| rt j | ndt j | dt j v st j
| rt j | nddz }dd|iz } t t j | dx}}y)uB 생성자에서 allowed_domains를 설정할 수 있어야 한다.Nr$ r& r
r' r* r+ z example.comzinsurance.co.krrL )r allowed_domainsr, )z7%(py2)s
{%(py2)s = %(py0)s.allowed_domains
} == %(py4)sr domainsrv rY rT ) r
r6 r7 r8 r9 r: r; r< r= r r )
r@ rA rB rC rD r r rE ra r^ s
r test_allowed_domains_setz0TestInsuranceSpiderInit.test_allowed_domains_set s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,, "34"22-. #
%%0%0000%000000v000v000%0000000000000000r c
d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j dg }|j }t | }d
}t | }||k( } | sgt j d | fd ||f d
t j v st j
t rt j t nd
dt j v st j
| rt j | ndt j | t j | d
t j v st j
t rt j t nd
t j | t j | dz }
dd|
iz } t t j | dx}x}x} x}}y)u7 output_dir 기본값은 'crawl_output'이어야 한다.Nr$ r& r
r' r* r+ rL r crawl_outputr, zp%(py5)s
{%(py5)s = %(py0)s(%(py3)s
{%(py3)s = %(py1)s.output_dir
})
} == %(py11)s
{%(py11)s = %(py7)s(%(py9)s)
}r r r( rS r) r+ r3 r5 r assert %(py13)spy13 r
r6 r7 r8 r9 r: r; r< r= r
output_dirr r@ rA rB rC rD r r] r @py_assert10rF r @py_format14s r test_output_dir_defaultz/TestInsuranceSpiderInit.test_output_dir_default sW )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,"22?T>UV%%>t%&>~>$~*>>&*>>>>>&*>>>>>>>t>>>t>>>>>>F>>>F>>>%>>>&>>>>>>$>>>$>>>~>>>*>>>>>>>>>r c
d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j dgd
}|j }t | }d }t | }||k( } | sgt j d | fd ||f d
t j v st j
t rt j t nd
dt j v st j
| rt j | ndt j | t j | d
t j v st j
t rt j t nd
t j | t j | dz }
dd|
iz } t t j | dx}x}x} x}}y)u= output_dir을 커스텀으로 설정할 수 있어야 한다.Nr$ r& r
r' r* r+ rL z/tmp/test_outputr r r, r r r r r r r r s r test_output_dir_customz.TestInsuranceSpiderInit.test_output_dir_custom
sa )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,"22-.)
%%Bt%&B/AB$/A*BB&*BBBBB&*BBBBBBBtBBBtBBBBBBFBBBFBBB%BBB&BBBBBB$BBB$BBB/ABBB*BBBBBBBBBr c * d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j dg }|j }d}||u }|st j d
|fd ||f d t j v st j
| rt j | nd t j | t j | d
z }dd|iz }t t j | dx}x}}y)u4 extraction_config 기본값은 None이어야 한다.Nr$ r& r
r' r* r+ rL r is)z9%(py2)s
{%(py2)s = %(py0)s.extraction_config
} is %(py5)sr r r r3 r
r6 r7 r8 r9 r: r; r< r= r extraction_configr s r #test_extraction_config_default_nonez;TestInsuranceSpiderInit.test_extraction_config_default_none s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,"22?T>UV''/4/'4////'4//////v///v///'///4///////r c d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}dd d
d id }t j d
g| }|j }||k( }|st j d|fd||f dt j v st j
| rt j | ndt j | dt j v st j
| rt j | nddz }dd|iz } t t j | dx}}y)uD extraction_config를 딕셔너리로 설정할 수 있어야 한다.Nr$ r& r
r' r* r+ css.productr .namemode css_selectorfieldsrL r r r, )z9%(py2)s
{%(py2)s = %(py0)s.extraction_config
} == %(py4)sr configrv rY rT r )
r@ rA rB rC rD r r rE ra r^ s
r test_extraction_config_setz2TestInsuranceSpiderInit.test_extraction_config_set s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,, PWGXY"22-. $
''1'61111'6111111v111v111'111111611161111111r r )
r r r r r r r r r r r r r rp r r r r s .L L
, , ) ) 1 1 ? ? C C 0 0 2 2r r c | e Zd ZdZed d Zed d Zed d Zed d Zed d Z ed d Z
y)
TestInsuranceSpiderParseu, InsuranceSpider.parse() 메서드 테스트.Nc d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}dd d
d d d
}t j dg| }t t d }t j t || }|D cg c] } t | t s| }
} t# |
}d} || k\ } | st j d| fd|| f dt j v st j
t" rt j t" nddt j v st j
|
rt j |
ndt j | t j | dz }
dd|
iz }t t j | dx}x} } g }d}|
d } || v } | }| s
d}|
d }||v }|}|st j d | fd || f t j | t j | d z }
d d|
iz }|j% | | s_t j d fd f t j | t j | d!z }d"d#|iz }|j% | t j& |d i z }d$d%|iz }t t j | dx}x}x}x} x} x}x}}yc c} w )&uQ CSS 모드에서 extraction_config 기반으로 아이템을 추출해야 한다.Nr$ r& r
r' r* r+ r r r z.pricer pricer rL r r
rN rk rm rP itemsrR rU rV r r r r{ )z%(py3)s in %(py6)s)r) rT z%(py8)s)z%(py11)s in %(py14)s)r py14z%(py16)spy16zassert %(py19)spy19)r
r6 r7 r8 r9 r: r; r< r= r r
HTML_PRODUCTSasynciorunr r dictrP r _format_boolop)r@ rA rB rC rD r r r r rr rG r] r^ r_ r` r @py_assert13 @py_assert12 @py_format15 @py_format17 @py_format18 @py_format20s r "test_parse_css_mode_extracts_itemsz;TestInsuranceSpiderParse.test_parse_css_mode_extracts_items2 s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,, & &:
#22-. $
&m9NO++ 3FH EF #;qz!T':;;5z Q zQ zQ s s 5 5 z Q 8v8q8v!8W8a8W%8888v888v8888888888W888W88888888888888888 table 모드에서 테이블 데이터를 추출해야 한다.Nr$ r& r
r' r* r+ table)r table_selectorrL r r rN rk rm rP r rR rU rV c 3 0 K | ] }d |v xs d|v yw)u 보험명col_0Nrp )rq r s r rs zATestInsuranceSpiderParse.test_parse_table_mode..X s! Ld;$&9'T/9Ls rt ru rv )r
r6 r7 r8 r9 r: r; r< r= r r
HTML_TABLEr r r r r rP ru )r@ rA rB rC rD r r r r r r rG r] r^ r_ rE ra s r test_parse_table_modez.TestInsuranceSpiderParse.test_parse_table_modeG s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,, %
#22-. $
&j6KL++ 3FH EF #;qz!T':;;5z Q zQ zQ s s 5 5 z Q LeLLsLLLLLLLLLsLLLsLLLLLLLLLLLLLL )-, ,,,, ,,,,,, ,,, ,,,,,,,,,, ", w'
#22-. $
&m9NO++ 3FH EF'4((((((((z(((z(((((('((('((((((4(((4((((((((((r c d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}dd d
d id }t j d
g| }t t d }t j t || }|D cg c] } t | t sd| v s| }
} t# |
}d} || k\ } | st j d| fd|| f dt j v st j
t" rt j t" nddt j v st j
|
rt j |
ndt j | t j | dz }
dd|
iz }t t j | dx}x} } |
d d }d}||k( }|slt j d|fd ||f t j | t j | d z }d d |iz }
t t j |
dx}x}}yc c} w ) uG 각 아이템에 _source_url 메타데이터가 첨부되어야 한다.Nr$ r& r
r' r* r+ r r r r r rL r https://example.com/productsr _source_urlrN rk rm rP r rR rU rV r r, rW rX rY rT r
r6 r7 r8 r9 r: r; r< r= r r r r r r r r rP )r@ rA rB rC rD r r r r r r rG r] r^ r_ r` rE ra s r #test_parse_adds_source_url_metadatazDDc d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j 5 }t j dg| }|j j dd
t j |j! d t j |j# t% t' | j) d
}t+ | }d}||k\ } | st j d| fd||f dt j v st j
t* rt j t* nddt j v st j
| rt j | ndt j | t j | dz }
dd|
iz } t t j | dx}x} }ddd y# 1 sw Y yxY w)u? on_close()는 ResponseHistory를 파일로 저장해야 한다.Nr$ r& r
r' r* r+ rL r rM Fr z *history*rN rk rm rP
history_filesrR rU rV )r
r6 r7 r8 r9 r: r; r< r= r r r response_historyr[ r r r on_closer r globrP ) r@ rA rB rC rD r r r rG r] r^ r_ s r test_on_close_saves_historyz4TestInsuranceSpiderHooks.test_on_close_saves_history s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,
(
(
* +f&66 12!F
# # * *+@# F KK7 8 KK) * f !2!2;!? @M}% * *%* * * *% * * * * * *3 * * *3 * * * * * *} * * *} * * *% * * * * * * * * * * + + +s
F0JJ c d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}dd l m
} t j d
g } |d
}t d }t j |j || y)
u: on_error()는 예외 없이 에러를 로깅해야 한다.Nr$ r& r
r' r* r+ r r
rL r zConnection failed)r
r6 r7 r8 r9 r: r; r< r= r r r ConnectionErrorr r on_error) r@ rA rB rC rD r r requesterrors r test_on_error_logs_errorz1TestInsuranceSpiderHooks.test_on_error_logs_error s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,-"22?T>UV/0 34 FOOGU34r c d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j dg }d
d d }t j |j | }||k( }|st j d
|fd||f dt j v st j
| rt j | nddt j v st j
| rt j | nddz }dd|iz } t t j | d}y)u5 유효한 아이템은 그대로 반환해야 한다.Nr$ r& r
r' r* r+ rL r 화재보험50000r r, )z%(py0)s == %(py2)srQ r )r( r1 zassert %(py4)sr2
r
r6 r7 r8 r9 r: r; r< r= r r r on_scraped_item)
r@ rA rB rC rD r r rQ @py_format3ra s
r &test_on_scraped_item_passes_valid_itemz?TestInsuranceSpiderHooks.test_on_scraped_item_passes_valid_item s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,"22?T>UV&9V 3 3D 9:~ v v v r c 8 d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j dg }dd
d }t j |j | }d}||u }|st j d |fd
||f dt j v st j
| rt j | ndt j | dz }dd|iz }t t j | dx}}y)uR 모든 값이 None이나 빈 문자열이면 None을 반환해야 한다 (드롭).Nr$ r& r
r' r* r+ rL r r r )z%(py0)s is %(py3)srQ r+ )r@ rA rB rC rD r
empty_itemrQ s r 'test_on_scraped_item_filters_empty_itemz@TestInsuranceSpiderHooks.test_on_scraped_item_filters_empty_item s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,"22?T>UV.2R%@
V 3 3J ?@ v~ v v v r c 8 d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j dg }d
dd }t j |j | }d}||u}|st j d|fd||f d t j v st j
| rt j | nd t j | dz }dd|iz }t t j | dx}}|d
}d
} || k( }|slt j d|fd|| f t j | t j | dz }
dd|
iz } t t j | dx}x}} y)uG 일부 값만 비어 있는 경우 아이템을 통과시켜야 한다.Nr$ r& r
r' r* r+ rL r r) r rQ r r, rW rX rY rT r+ ) r@ rA rB rC rD r r rQ r` rE ra r^ s r )test_on_scraped_item_partial_empty_passeszBTestInsuranceSpiderHooks.test_on_scraped_item_partial_empty_passes s? )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,"22?T>UV&6V 3 3D 9: !!vT!!!!vT!!!!!!v!!!v!!!T!!!!!!!f~/ /~ ////~ ///~/// ///////r r ) r r r r r r r r! r' r. r2 r4 rp r r r r s 2
'
' 8 8 + + 5 5 0 0r r c X e Zd ZdZedd Zedd Zedd Zedd Zy)TestInsuranceSpiderRunu< InsuranceSpider.run() 메서드 및 출력 형식 테스트.Nc d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}dd l m
}m }m } t j d
g } | |d d
|ddig } t j |d| 5 }
|j! } |
j# t% | | } | sddt j v st j
t$ rt j t$ nddt j v st j
| rt j | nddt j v st j
| rt j | ndt j | dz }
t t j |
d} ddd y# 1 sw Y yxY w)uD run()은 start()를 호출하고 CrawlResult를 반환해야 한다.Nr$ r& r
r' r* r+ r CrawlResult
CrawlStatsItemListrL r rN rh requests_count
items_scrapedr u 보험Astatsr start return_valuer r rQ r9 r )r
r6 r7 r8 r9 r: r; r< r= scrapling.spiders.resultr9 r: r; r r objectr assert_called_oncer )r@ rA rB rC rD r9 r: r; r mock_result
mock_startrQ rE ra s r -test_run_calls_start_and_returns_crawl_resultzDTestInsuranceSpiderRun.test_run_calls_start_and_returns_crawl_result sg )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,NN"22?T>UV! AQ?VY /01
\\&'
D 3
ZZ\F ) ) + fk2 22 2 2 2 2 2: 2 2 2: 2 2 2 2 2 2f 2 2 2f 2 2 2 2 2 2k 2 2 2k 2 2 22 2 2 2 2 2 2 3 3 3s
D5IIc * d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}dd l m
}m }m } t j 5 }t j d
g| } |d d
id dig }
| |dd |
} t! j" | d| 5 | j% d ddd t' t) | j+ d } t- | }d}
||
k\ }|st j d|fd||
f dt j v st j
t, rt j t, nddt j v st j
| rt j | ndt j | t j |
d z }d d |iz }t t j | dx}x}}
ddd y# 1 sw Y FxY w# 1 sw Y yxY w) uB run(output_format='jsonl')은 JSONL 파일로 내보내야 한다.Nr$ r& r
r' r* r+ r r8 rL r r r) u 자동차보험rh r< r? rA rB jsonl
output_formatz*.jsonlrN rk rm rP jsonl_filesrR rU rV r
r6 r7 r8 r9 r: r; r< r= rD r9 r: r; r r r r rE r r r r rP )r@ rA rB rC rD r9 r: r; r r
mock_itemsrG rN rG r] r^ r_ s r test_run_exports_jsonlz-TestInsuranceSpiderRun.test_run_exports_jsonl s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,NN
(
(
* )f&66 12!F "FN#;fFW=X"YZJ% C K fgKH
2
1
2 tF| 0 0 ;JJc H d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j j dd gd
d d d
}t |t }|s ddt j v st j
t rt j t nddt j v st j
| rt j | nddt j v st j
t rt j t ndt j | dz }t t j | d}|d }d}||k( }|slt j d|fd||f t j | t j | dz }dd|iz } t t j | dx}x}}|d }d g}||k( }|slt j d|fd||f t j | t j | dz }dd|iz } t t j | dx}x}}|d }d
}||k( }|slt j d|fd||f t j | t j | dz }dd|iz } t t j | dx}x}}d }||v }|st j d |fd ||f t j | dt j v st j
| rt j | ndd z }dd|iz }t t j | dx}}y) uZ create_cron_config()는 cokacdir --cron 연동 설정 딕셔너리를 반환해야 한다.Nr$ r& r
r' r* r+ z 0 6 * * *z https://example.com/insurancez/data/insurancer r )r r )scheduler r r r r r r r rV r, rW rX rY rT r r r r{ r} r~ )
r
r6 r7 r8 r9 r: r; r< r= r create_cron_configr r )
r@ rA rB rC rD r rE ra r` r^ s
r test_create_cron_configz.TestInsuranceSpiderRun.test_create_cron_configB sx )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,"22EE 78(',j I F
&$''''''''z'''z''''''&'''&''''''$'''$''''''''''j!0[0![0000![000!000[0000000l#H(G'HH#'HHHHH#'HHHH#HHH'HHHHHHHHl#8'88#'88888#'8888#888'88888888","f,,,,"f,,,",,,,,,f,,,f,,,,,,,r r ) r r r r r rI rQ rT rX rp r r r6 r6 sS F
3
3 ) )* ( ((
-
-r r6 c X e Zd ZdZedd Zedd Zedd Zedd Zy)TestCheckpointIntegrationu& Checkpoint(crawldir) 통합 테스트.Nc d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j 5 }t | dz }t j d gt |
}|j }||k( }|st j d |fd ||f d
t j v st j
| rt j | nd
t j | dt j v st j
| rt j | nddz } dd| iz }
t t j |
dx}}ddd y# 1 sw Y yxY w)uC crawldir을 지정하면 Spider.crawldir에 반영되어야 한다.Nr$ r& r
r' r* r+ checkpointsrL )r crawldirr, )z0%(py2)s
{%(py2)s = %(py0)s.crawldir
} == %(py4)sr r] rv rY rT )r
r6 r7 r8 r9 r: r; r< r= r r r r r r] ) r@ rA rB rC rD r r] r rE ra r^ s r test_crawldir_is_setz.TestCheckpointIntegration.test_crawldir_is_set[ s- )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,
(
(
* /fF|m3H&66 12 X F ?? .?h. . . .?h . . . . . .6 . . .6 . . .? . . . . . .h . . .h . . . . . . .
/ / /s
D!G44G=c * d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j dg }|j }d}||u }|st j d
|fd ||f d t j v st j
| rt j | nd t j | t j | d
z }dd|iz }t t j | dx}x}}y)u? crawldir 미지정 시 Spider.crawldir이 None이어야 한다.Nr$ r& r
r' r* r+ rL r r )z0%(py2)s
{%(py2)s = %(py0)s.crawldir
} is %(py5)sr r r r3 ) r
r6 r7 r8 r9 r: r; r< r= r r] r s r test_crawldir_none_by_defaultz7TestCheckpointIntegration.test_crawldir_none_by_defaultg s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,"22?T>UV&$&$&&&&$&&&&&&v&&&v&&&&&&$&&&&&&&r c . d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j dgd
}|j }d }||k( }|st j d |fd ||f d
t j v st j
| rt j | nd
t j | t j | dz }dd|iz }t t j | dx}x}}y)uD interval 파라미터가 _interval 속성에 반영되어야 한다.Nr$ r& r
r' r* r+ rL g @)r intervalr, z1%(py2)s
{%(py2)s = %(py0)s._interval
} == %(py5)sr r r r3 r
r6 r7 r8 r9 r: r; r< r= r _intervalr s r test_checkpoint_interval_setz6TestCheckpointIntegration.test_checkpoint_interval_setn s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,"22-.
(5( 5(((( 5((((((v(((v((( (((5(((((((r c , d}t |u}|st j d|fdt |f dt j v st j
t rt j t ndt j | dz }dd|iz }t t j | dx}}t j dg }|j }d
}||k( }|st j d |fd ||f d
t j v st j
| rt j | nd
t j | t j | dz }dd|iz }t t j | dx}x}}y)u, interval 기본값은 300.0이어야 한다.Nr$ r& r
r' r* r+ rL r g r@r, rc r r r r3 rd r s r test_checkpoint_interval_defaultz:TestCheckpointIntegration.test_checkpoint_interval_defaultx s )-, ,,,, ,,,,,, ,,, ,,,,,,,,,,"22?T>UV (5( 5(((( 5((((((v(((v((( (((5(((((((r r ) r r r r r r^ r` rf rh rp r r rZ rZ X sS 0 / / ' ' ) ) ) )r rZ )rL rM )1r builtinsr8 _pytest.assertion.rewrite assertionrewriter6 r r sysr pathlibr typesr typingr r
unittest.mockr r r pytestr insertr __file__parentr
__annotations__r ModuleNotFoundError_MISSINGmarkskipifr r r HTML_EMPTY_ITEMSintr r r r" r r r r6 rZ rp r r r} sb
5 5
3tH~,,334 5 +/ 8J' . 0 $;;%% U &
(
%
" s tCy c+ c+VF2 F2\q q rT0 T0xL- L-h%) %)s s
D DD