Difference between chunk size and page size Spring Batch

Andrew :

I have a spring batch job in which I set the chunk size to 1000 and the reader in that job is a JpaPagingItemReader.

In the reader I set the page size to 20. Does this mean that every chuck of 1000 items read the reader must take out of the db 20 items?

If not what is the difference between them?

Sabir Khan :

With your current configuration, if your every read item makes to writer ( i.e. if it doesn't get filtered out in processor ) then you will need 1000/20 = 50 database reads to reach a chunk size i.e. when you actually call the writer for writing.

Spring Batch holds processed items in memory till you reach a chunk size and holding items costs memory.

Your current configuration is holding data in memory and making unnecessary database calls while we wish to reduce both of those things.

So your configuration needs to be reverse of what you are doing i.e. increase reader page size to a minimum equal to chunk size / commit interval or more so data once read gets processed in small small chunks then you go again and read database .

So as you have noticed till current write up, conceptually these are unrelated concepts - reader page size is to minimize database calls ( and this concept is not a spring batch concept but reader specific - if its not a paging reader, this concept doesn't come into picture ) while chunk size is about committing processed data in small small chunks to reduce memory foot print.

이 기사는 인터넷에서 수집됩니다. 재 인쇄 할 때 출처를 알려주십시오.

침해가 발생한 경우 연락 주시기 바랍니다[email protected] 삭제

에서 수정
0

몇 마디 만하겠습니다

0리뷰
로그인참여 후 검토

관련 기사

Difference between size after resize in ImagemagicK

HTTP 1.1 chunk-size computation algorithm

What is the difference between setting window size using Options and setting window size using set_window_size method

Symfony HttpClient Stream and chunk->getContent() Body size limit exceeded

Page size changing when printing

In Adobe Acrobat, what is the difference between Save As > "Reduce Size PDF" vs Save As > "Optimized PDF"

Accumulating gradients for a larger batch size with PyTorch

hibernate.jdbc.batch_size 인쇄

How to change the batch size during training?

PyMongo 커서 batch_size

CNN의 batch_size 이해

Why does Keras LSTM batch size used for prediction have to be the same as fitting batch size?

Size difference copying from NFTS to HFS+

Font size in ioslides markdown presentation title page

Keras CNN : 모든 batch_size> 1 인 호환되지 않는 모양 [batch_size * 2,1] 대 [batch_size, 1]

Spring Batch에서 Step, Tasklet 및 Chunk의 차이점

Keras : batch_size가있는 TypeError

pmap () Julia에서 batch_size 설정

TensorFlow DataSet `from_generator` with variable batch size

tensorflow의 Batch_size? 개념 이해

Exporting big google sheets file as multiple csv with batch size 500

호출 함수의 가변 batch_size

How do I format man page for printing on custom page size?

train_data.cache (). shuffle (BUFFER_SIZE) .batch (BATCH_SIZE) .repeat ()는 무엇을합니까?

tf model.fit ()의 batch_size 대 tf.data.Dataset의 batch_size

대상 batch_size (6)와 일치하는 예상 입력 batch_size (18)

PyTorch : 대상 batch_size (64)와 일치하는 예상 입력 batch_size (12)

Why is the difference in file size and it's size on disk bigger than 4 KiB?

Tensorflow 및 Batch Normalization with Batch Size == 1 => 모두 0을 출력합니다.

TOP 리스트

뜨겁다태그

보관