File: //lib64/python3.9/site-packages/borg/__pycache__/cache.cpython-39.opt-1.pyc
a
HZ�hs� � @ s. d dl Z d dlZd dlZd dlZd dlZd dlmZ d dlmZ d dl m
Z
ddl m Z e � Z e d�Z
ddlmZmZ dd lmZmZmZ dd
lmZ dd lmZ dd lmZ dd
lmZmZ ddlmZm Z m Z m Z ddlm Z ddlm Z ddlm!Z! ddlm"Z" ddlm#Z#m$Z$ ddlm%Z%m&Z& ddlm'Z' ddlm(Z( ddl)m*Z*m+Z+ ddl,m-Z- ddl.m/Z/m0Z0m1Z1 ddl2m3Z3 ddl4m5Z5 dd l6m7Z7 dd l8m9Z9 ed d �Z:G d d!� d!�Z;d"d#� Z<d$d%� Z=d6d&d'�Z>d(d)� Z?d*d+� Z@G d,d-� d-�ZAG d.d/� d/�ZBG d0d1� d1�ZCG d2d3� d3eC�ZDG d4d5� d5eC�ZEdS )7� N)� unhexlify)�
namedtuple)� perf_counter� )�
create_loggerzborg.debug.files_cache)� CACHE_README�FILES_CACHE_MODE_DISABLED)�
ChunkIndex�ChunkIndexEntry�CacheSynchronizer)�Location)�Error)�Manifest)�
get_cache_dir�get_security_dir)�
int_to_bigint�
bigint_to_int�
bin_to_hex�parse_stringified_list)�format_file_size)�safe_ns)�yes)�remove_surrogates)�ProgressIndicatorPercent�ProgressIndicatorMessage)�set_ec� EXIT_WARNING)� safe_unlink)�msgpack)� ArchiveItem�ChunkListEntry)� PlaintextKey)�IntegrityCheckedFile� DetachedIntegrityCheckedFile�FileIntegrityError)�Lock)�SaveFile)�cache_if_remote)�LIST_SCAN_LIMIT�FileCacheEntryz age inode size cmtime chunk_idsc @ s� e Zd ZdZdd� Zeddd��Zdd� Zd d
� Zd d � Z d d
d�Z
d dd�Z d dd�Z dddd�dd�Z
d dd�Zdd� ZdS ) �SecurityManageras
Tracks repositories. Ensures that nothing bad happens (repository swaps,
replay attacks, unknown repositories etc.).
This is complicated by the Cache being initially used for this, while
only some commands actually use the Cache, which meant that other commands
did not perform these checks.
Further complications were created by the Cache being a cache, so it
could be legitimately deleted, which is annoying because Borg didn't
recognize repositories after that.
Therefore a second location, the security database (see get_security_dir),
was introduced which stores this information. However, this means that
the code has to deal with a cache existing but no security DB entry,
or inconsistencies between the security DB and the cache which have to
be reconciled, and also with no cache existing but a security DB entry.
c C sV || _ t|j�| _t|�| _tj�| jd�| _tj�| jd�| _ tj�| jd�| _
d S )Nzkey-type�locationzmanifest-timestamp) �
repositoryr �id_str�dir� cache_dir�os�path�join�
key_type_file�
location_file�manifest_ts_file)�selfr, � r7 �0/usr/lib64/python3.9/site-packages/borg/cache.py�__init__? s
zSecurityManager.__init__Nc C s( |p t | j�}tj�|�r$t�|� dS )z:destroy the security dir for ``repository`` or at ``path``N)r r- r0 r1 �exists�shutil�rmtree�r, r1 r7 r7 r8 �destroyG s zSecurityManager.destroyc C s t dd� | j| j| jfD ��S )Nc s s | ]}t j�|�V qd S �N)r0 r1 r: )�.0�fr7 r7 r8 � <genexpr>O s �z(SecurityManager.known.<locals>.<genexpr>)�allr3 r4 r5 �r6 r7 r7 r8 �knownN s �zSecurityManager.knownc
C s� | � � s dS zFt| j��(}|�� }|t|j�kW d � W S 1 sF0 Y W n0 ty� } zt�d|� W Y d }~n
d }~0 0 d S )NFz&Could not read/parse key type file: %s) rE �openr3 �read�str�TYPE�OSError�logger�warning)r6 �key�fd�type�excr7 r7 r8 � key_matchesR s 2zSecurityManager.key_matchesc C s� t �d| jj| j� | jj�� }t �d|� t �dt|j�� t �d|j � t
| j ��}|� |� W d � n1 sv0 Y t
| j
�� }|� t|j�� W d � n1 s�0 Y t
| j�� }|� |j � W d � n1 s�0 Y d S )Nz#security: saving state for %s to %sz security: current location %sz security: key type %sz security: manifest timestamp %s)rK �debugr, r- r. � _location�canonical_pathrH rI � timestampr& r4 �writer3 r5 )r6 �manifestrM Zcurrent_locationrN r7 r7 r8 �save\ s ( . zSecurityManager.savec
C sZ zBt | j��}|�� }W d � n1 s*0 Y t�d|� W nV tyf t�d| j� d }Y n4 ty� } z t�d|� d }W Y d }~n
d }~0 0 |r�|jr�||jkr�|j}t�d|� | j j
� � }|�rV||k�rVd� ||�d }t
|ddd d
d ��st�� �t�d � t| j��}|�|� W d � n1 �s>0 Y |�rV|�� d S )
Nz#security: read previous location %rz-security: previous location file %s not found�)Could not read previous location file: %sz.security: using previous_location of cache: %rzDWarning: The repository at location {} was previously located at {}
z Do you want to continue? [yN] � Aborting.�Invalid answer, aborting.FZ BORG_RELOCATED_REPO_ACCESS_IS_OK�Z false_msgZ invalid_msgZretryZenv_var_overridez<security: updating location stored in cache and security dir)rF r4 rG rK rR �FileNotFoundErrorrJ rL �previous_locationr, rS rT �formatr �Cache�RepositoryAccessAbortedr& rV rX )r6 � cache_configrN r^ rP Zrepository_location�msgr7 r7 r8 �assert_location_matchesi s: & ���
*z'SecurityManager.assert_location_matchesc
C s� zBt | j��}|�� }W d � n1 s*0 Y t�d|� W nV tyf t�d| j� d}Y n4 ty� } z t�d|� d}W Y d }~n
d }~0 0 |r�t||j p�d�}t�d|� |r�||j kr�t
|t �r�t �
� �nt �� �d S )Nz$security: read manifest timestamp %rz.security: manifest timestamp file %s not found� rY z4security: determined newest manifest timestamp as %s)rF r5 rG rK rR r] rJ rL �maxrU �
isinstancer! r` �RepositoryIDNotUnique�RepositoryReplay)r6 rW rM rb rN rU rP r7 r7 r8 �assert_no_manifest_replay� s" &
z)SecurityManager.assert_no_manifest_replayc C sD |r&|j d ur&|j t|j�kr&t�� �| �� r@| �|�s@t�� �d S r? )�key_typerH rI r` �EncryptionMethodMismatchrE rQ )r6 rM rb r7 r7 r8 �assert_key_type� s z SecurityManager.assert_key_typeT)rb �warn_if_unencrypted� lock_waitc C s� | � |||� |r"| �|||� nVt| j|d�}|�� rl|� | �|||� W d � qx1 s`0 Y n | �||� t�d� d S )N�ro z/security: repository checks ok, allowing access)�assert_access_unknown�_assert_secure� CacheConfigr, r: rK rR )r6 rW rM rb rn ro r7 r7 r8 �
assert_secure� s . z SecurityManager.assert_securec C sF | � |� | �||� | �|||� | �� sBt�d� | �||� d S )Nz3security: remembering previously unknown repository)rd rm rj rE rK rR rX )r6 rW rM rb r7 r7 r8 rr � s
z SecurityManager._assert_securec C sb |j s^| �� s^d}| p(t|ddddd�}|rV|r>t�d� n
t�d� | �||� nt�� �d S ) NziWarning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] rZ r[ FZ*BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OKr\ zIsecurity: remembering unknown unencrypted repository (explicitly allowed)z-security: initializing unencrypted repository)Zlogically_encryptedrE r rK rR rX r` �CacheInitAbortedError)r6 rn rW rM rc Z allow_accessr7 r7 r8 rq � s �
z%SecurityManager.assert_access_unknown)N)N)N)N)N)�__name__�
__module__� __qualname__�__doc__r9 � staticmethodr> rE rQ rX rd rj rm rt rr rq r7 r7 r7 r8 r* + s
r* c C s t | �}|j||j|d� d S )Nrp )r* rt rM )r, rW ro Zsmr7 r7 r8 rt � s rt c C s� |j �� }t|�}t| �}|j|jkr�|j|jkr�|j|jkr�|j|jkr�|jr�|jr�|j�d�r�|j�d�r�|jdd � |jdd � kr�|S | S d S )Nz/~/z/./� ) rS rT r �proto�user�host�portr1 �
startswith)Zcache_locationr, Z
repo_locationZrlZclr7 r7 r8 � recanonicalize_relative_location� s
0��
�
��r� c C s |pt j�t� | j�S r? )r0 r1 r2 r r- r= r7 r7 r8 r/ � s r/ c C s t j�dd�} | rd| S dS )NZBORG_FILES_CACHE_SUFFIXre �files.�files)r0 �environ�get��suffixr7 r7 r8 �files_cache_name� s r� c C s dd� t �| �D �d S )Nc S s" g | ]}|d ks|� d�r|�qS )r� r� )r� �r@ �fnr7 r7 r8 �
<listcomp>� � z-discover_files_cache_name.<locals>.<listcomp>r )r0 �listdir)r1 r7 r7 r8 �discover_files_cache_name� s r� c @ s` e Zd Zddd�Zdd� Zdd� Zdd � Zd
d � Zd d
� Zdd� Z ddd�Z
dd� Z dd� Z dS )rs Nc C s4 || _ t||�| _tj�| jd�| _d | _|| _d S )N�config)r, r/ r1 r0 r2 � config_path�lockro )r6 r, r1 ro r7 r7 r8 r9 � s
zCacheConfig.__init__c C s | � � | S r? )rF rD r7 r7 r8 � __enter__� s zCacheConfig.__enter__c C s | � � d S r? ��close�r6 �exc_typeZexc_valZexc_tbr7 r7 r8 �__exit__� s zCacheConfig.__exit__c C s t j�| j�S r? )r0 r1 r: r� rD r7 r7 r8 r: s zCacheConfig.existsc C s� t jd d�}|�d� |�ddd� |�dd| jj� |�ddd� |�d� |�ddd� t| j��}|�|� W d � n1 s�0 Y d S ) N�Z
interpolation�cache�version�1r, rW re � integrity) � configparser� ConfigParser� add_section�setr, r- r& r� rV )r6 r� rN r7 r7 r8 �create s
zCacheConfig.createc C s. t tj�| jd�d| jd��� | _| �� d S )Nr� T)� exclusive�timeout)r% r0 r1 r2 ro �acquirer� �loadrD r7 r7 r8 rF s "zCacheConfig.openc C s� t jd d�| _t| j�� }| j�|� W d � n1 s:0 Y | �| j� | j�dd�| _t | j�dd��| _
| jjddd d�| _ | jjddd d�| _ t
t| jjddd d���| _t
t| jjdd
d d���| _zBt| j�d ��| _| j�dd�| j�d�k�ri | _t�d � W n& t j�y< t�d
� i | _Y n0 | jjddd d�}|�rft|| j�| _nd | _| j�
dd| jj � � � d S )Nr� r� r, rW rU )�fallbackrk �ignored_featuresre �mandatory_featuresr� zHCache integrity data not available: old Borg version modified the cache.zTCache integrity: No integrity data found (files, chunks). Cache is from old version.r^ ) r� r� �_configrF r� Z read_file�_check_upgrader� �idr � manifest_idrU rk r� r r� r� �dict�itemsr� �poprK rL �NoSectionErrorrR r� r, r^ rS rT )r6 rN r^ r7 r7 r8 r� s. *
zCacheConfig.loadc C s� |r�| j �dd|j� | j �dd|j� | j �ddd�| j�� | j �ddd�| j�� | j �d�sp| j �d� | j �
� D ]\}}| j �d||� qz| j �dd|j� |r�| j �ddt |j �� t
| j�� }| j �|� W d � n1 s�0 Y d S ) Nr� rW rU r� �,r� r� rk )r� r� r- rU r2 r� r� Z has_sectionr� r� r� rH rI r& r� rV )r6 rW rM �file�integrity_datarN r7 r7 r8 rX 3 s zCacheConfig.savec C s | j d ur| j �� d | _ d S r? )r� �releaserD r7 r7 r8 r� C s
zCacheConfig.closec C sh z8| j �dd�}d}||kr6| �� td|||f ��W n* tjyb | �� td| �d �Y n0 d S )Nr� r� r z0%s has unexpected cache version %d (wanted: %d).z#%s does not look like a Borg cache.)r� Zgetintr� � Exceptionr� r� )r6 r� Z
cache_versionZwanted_versionr7 r7 r8 r� H s �
zCacheConfig._check_upgrade)NN)NN)
rv rw rx r9 r� r� r: r� rF r� rX r� r� r7 r7 r7 r8 rs � s
rs c @ s� e Zd ZdZG dd� de�ZG dd� de�ZG dd� de�ZG dd � d e�ZG d
d � d e�Z e
dd
d��Z e
ddd��Z d dddd de
ddf dd�Zd S )r` zClient Side cache
c @ s e Zd ZdZdS )zCache.RepositoryIDNotUniquez`Cache is newer than repository - do you have multiple, independently updated repos with same ID?N�rv rw rx ry r7 r7 r7 r8 rh X s rh c @ s e Zd ZdZdS )zCache.RepositoryReplayz�Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)Nr� r7 r7 r7 r8 ri [ s ri c @ s e Zd ZdZdS )zCache.CacheInitAbortedErrorz Cache initialization abortedNr� r7 r7 r7 r8 ru ^ s ru c @ s e Zd ZdZdS )z Cache.RepositoryAccessAbortedzRepository access abortedNr� r7 r7 r7 r8 ra a s ra c @ s e Zd ZdZdS )z Cache.EncryptionMethodMismatchzLRepository encryption method changed since last access, refusing to continueNr� r7 r7 r7 r8 rl d s rl Nc C s( t | |�}ttj�|d�dd��� d S )Nr� T)r� )r/ r% r0 r1 r2 �
break_lockr= r7 r7 r8 r� g s
zCache.break_lockc C sH |pt j�t� | j�}t j�|d�}t j�|�rDt �|� t�|� dS )z3destroy the cache for ``repository`` or at ``path``r� N) r0 r1 r2 r r- r: �remover; r< )r, r1 r� r7 r7 r8 r> l s
z
Cache.destroyTFc
s� � ��������� �
f dd�}
������fdd�}| s@|
� S t ����}|�� r�|� |j�jk}W d � n1 sz0 Y |r�t�d� |
� S t�d� |� S )Nc
s t ����� �
���� �d� S )N) r, rM rW r1 �syncrn �progress�iecro �
cache_mode�consider_part_files)�
LocalCacher7 � r� r� r� rM ro rW r1 r� r, r� rn r7 r8 �localy s �z Cache.__new__.<locals>.localc s t ������ d�S )N)r, rM rW ro r� r� )�
AdHocCacher7 )r� r� rM ro rW r, r7 r8 �adhoc~ s �z Cache.__new__.<locals>.adhocz%Cache: choosing local cache (in sync)zKCache: choosing ad-hoc cache (local cache does not exist or is not in sync))rs r: r� r� rK rR )�clsr, rM rW r1 r� rn r� ro Zpermit_adhoc_cacher� r� r� r� r� rb Z
cache_in_syncr7 r� r8 �__new__u s *
z
Cache.__new__)N)N)rv rw rx ry r
rh ri ru ra rl rz r� r> r r� r7 r7 r7 r8 r` U s �r` c @ sH e Zd ZdZddd�Zdd� Zedg d��Zd d
� Zd d � Z d
d� Z
dS )�CacheStatsMixinz�All archives: {0.total_size:>20s} {0.total_csize:>20s} {0.unique_csize:>20s}
Unique chunks Total chunks
Chunk index: {0.total_unique_chunks:20d} {0.total_chunks:20d}Fc C s || _ i | _d S r? )r� �
pre12_meta)r6 r� r7 r7 r8 r9 � s zCacheStatsMixin.__init__c C s | j �| �� �S r? )�
str_formatr_ � format_tuplerD r7 r7 r8 �__str__� s zCacheStatsMixin.__str__�Summary)�
total_size� total_csize� unique_size� unique_csize�total_unique_chunks� total_chunksc C s� ddl m} | j�� \}}}}}}d\}}| jjD ]@}|| j| j| j|| jd�} | j | dd�}
||
j
7 }||
j 7 }q2| � ||||||��
� }
|
S )Nr )�Archive)r r )r� F)Z want_unique)�archiver� �chunksZ summarizerW �archivesr, rM r� Z
calc_statsZosize�csizer� �_asdict) r6 r� r� r� r� r� r� r� � archive_namer� �statsr7 r7 r8 r� � s �
�zCacheStatsMixin.statsc C s8 | � � }dD ]}t|| | jd�||< q | jf i |��S )N)r� r� r� �r� )r� r r� r� )r6 r� Zfieldr7 r7 r8 r� � s z CacheStatsMixin.format_tuplec C s | � � d S )Nr� )r� rD r7 r7 r8 �chunks_stored_size� s z"CacheStatsMixin.chunks_stored_sizeN)F) rv rw rx r� r9 r� r r� r� r� r� r7 r7 r7 r8 r� � s
r� c @ s� e Zd ZdZdddddeddfdd�Zdd� Zd d
� Zd d � Zd
d� Z dd� Z
dd� Z dd� Z dd� Z
dd� Zdd� Zdd � Zd d � Zd d � Zd!d"� Zd/d#d$�Zd0d%d&�Zd1d'd(�Zd2d)d*�Zd+d,� Zd-d.� ZdS )3r� z0
Persistent, local (client-side) cache.
NTFc C s t j| | d� || _|| _|| _|| _| | _|
| _d| _d| _ t
||�| _ t |�| _
t| j| j |�| _tj �| j �s�| j
�|||� | �� | �� zT| j
j||| jd� | �� s�| �� | �� |r�| jj| jjkr�| �� | � � W n | � � � Y n0 dS )aO
:param warn_if_unencrypted: print warning if accessing unknown unencrypted repository
:param lock_wait: timeout for lock acquisition (int [s] or None [wait forever])
:param sync: do :meth:`.sync`
:param cache_mode: what shall be compared in the file stat infos vs. cached stat infos comparison
r� NF)rb ) r� r9 r, rM rW r� r� r� rU �
txn_activer/ r1 r* �security_managerrs rb r0 r: rq r� rF rt �check_cache_compatibility�
wipe_cache�update_compatibilityr� r� r� �commitr� ) r6 r, rM rW r1 r� rn r� ro r� r� r� r7 r7 r8 r9 � s6
zLocalCache.__init__c C s | S r? r7 rD r7 r7 r8 r� � s zLocalCache.__enter__c C s | � � d S r? r� r� r7 r7 r8 r� � s zLocalCache.__exit__c C s t �| j� tt j�| jd�d��}|�t� W d � n1 sB0 Y | j�� t � �t j�| jd�� t �t j�| jd�� t
t j�| jt � �dd�� W d � n1 s�0 Y t
t j�| jd�d d��"}t j
| j|d
d � W d � n1 s�0 Y dS ) z0Create a new empty cache at `self.path`
ZREADME�wNr� �chunks.archive.dT�Zbinary�
pre12-metaF� ��indent)r0 �makedirsr1 rF r2 rV r rb r� r r& r� �json�dumpr� �r6 rN r7 r7 r8 r� � s (
zLocalCache.createc C s� | j �� ttj�| jd�d| j j�d�d�� }t� |�| _
W d � n1 sP0 Y d| j v rld | _ n| �
� zDttj�| jd��� }t�|�| _W d � n1 s�0 Y W n ttjfy� Y n0 d S )Nr� F�r1 rV r� �dr� )rb r� r" r0 r1 r2 r� r� r rG r� r� r� � _read_filesrF r� r� r] ZJSONDecodeErrorr� r7 r7 r8 �_do_open s
�*
.zLocalCache._do_openc C s2 t j�| j�s td| j ��| j�� | �� d S )Nz"%s Does not look like a Borg cache)r0 r1 �isdirr� rb rF �rollbackrD r7 r7 r8 rF s
zLocalCache.openc C sf t tj�| jd�d��"}tj| j|dd� W d � n1 s>0 Y | jd urb| j�� d | _d S )Nr� r� r� r� ) rF r0 r1 r2 r� r� r� rb r� r� r7 r7 r8 r� s
0
zLocalCache.closec
C s� i | _ d | _t�d� t�d� d }z�ttj�| jt � �d| j
j � t � �d���}t
jdd�}|�d�}|slq�|�|� z8|D ].\}}t|� }t
�|j|jd d ��| j |< q|W q\ ttfy� } z$d
t|� }W Y d }~q�W Y d }~q\d }~0 0 q\W d � n1 �s0 Y W nb t�yF } zd t|� }W Y d }~n:d }~0 t�yv } zd t| � }W Y d } ~ n
d } ~ 0 0 |d u�r�t�|� t�d
� i | _ t�dt| j �� d S )NzReading files cache ...z FILES-CACHE-LOAD: starting...Fr� T)Zuse_listi r )�agez#The files cache seems invalid. [%s]z#The files cache can't be read. [%s]z"The files cache is corrupted. [%s]z:Continuing without files cache - expect lower performance.z.FILES-CACHE-LOAD: finished, %d entries loaded.) r� �_newest_cmtimerK rR �files_cache_loggerr" r0 r1 r2 r� rb r� r� r ZUnpackerrG �feedr) �packb�_replacer� � TypeError�
ValueErrorrH rJ r$ rL �len)
r6 rc rN �u�data� path_hash�item�entryrP �fier7 r7 r8 r� ( s>
�
$ H "
zLocalCache._read_filesc C s t dd�}tj�| jd�}t�|� |�d� t�tj�| jd�|� |�d� t�tj�| jd�|� |�d� z t�tj�| jt� �|� W nJ t y� t
tj�|t� �d d
�� W d � n1 s�0 Y Y n0 t� tj�| jd�tj�| jd �� d | _ |�
� d S ) Nzcache.begin_transaction��msgid�txn.tmpz.Initializing cache transaction: Reading configr� z.Initializing cache transaction: Reading chunksr� z-Initializing cache transaction: Reading filesTr� �
txn.active)r r0 r1 r2 �mkdir�outputr; �copyr� r] r& �renamer� �finish)r6 �pi�txn_dirr7 r7 r8 � begin_txnI s$
$�zLocalCache.begin_txnc C s | j s
dS | j�| j| j� tdd�}| jdu�rF| jdu rBd| _tt j
� dd��}|� d� t
�d� tt j�| jt� �d d
��|}d }| j�� D ]\\}}tt�|�� }|jd kr�t|j�| jk s�|jd kr�|j|k r�t�||f|� |d 7 }q�W d � n1 �s0 Y t
�d
|� t
�d| j� t
�d|� |j| j j t� < |� d� tt j�| jd�d d
�� }| j � |� W d � n1 �s�0 Y |j| j j d< |� d� | j �| j| j� t � t j�| jd�t j�| jd�� t!�"t j�| jd�� d| _ |�#� dS )zCommit transaction
Nz cache.commitr l ���� ZBORG_FILES_CACHE_TTL� zSaving files cachez FILES-CACHE-SAVE: starting...T�r1 rV r r z>FILES-CACHE-KILL: removed all old entries with age >= TTL [%d]zCFILES-CACHE-KILL: removed all current entries with newest cmtime %dz7FILES-CACHE-SAVE: finished, %d remaining entries saved.zSaving chunks cacher� zSaving cache configr r F)$r� r� rX rW rM r r� r� �intr0 r� r� r r� rR r" r1 r2 r� r� r) r �unpackbr� r �cmtime�packr� rb r� r� rV r r; r< r )r6 r
ZttlrN Z entry_countr� r� r� r7 r7 r8 r� ] sJ
��*
,
�zLocalCache.commitc C s� t j�t j�| jd��r.t�t j�| jd�� t j�| jd�}t j�|�r�t�t j�|d�| j� t�t j�|d�| j� t�t j�|t|��| j� t �|t j�| jd�� t j�t j�| jd��r�t�t j�| jd�� d| _ | �
� dS )z3Roll back partial and aborted transactions
r r r� r� FN) r0 r1 r: r2 r; r< r r� r r� r� )r6 r r7 r7 r8 r� � s zLocalCache.rollbackc s` t j��jd�� t� �d�
d�d�d�
d� d � fdd� � � �fdd���fdd ���fd
d ��d � fd
d� ���� �
fdd����
���fdd���� fdd���� �fdd���fdd�� ������� �
� �
����fdd�}� �fdd �}�j�tjjf� �� � t
�j �j d ��0�|� t j�
� ��_|�j��_W d � n1 �sR0 Y d S )!a� Re-synchronize chunks cache with repository.
Maintains a directory with known backup archive indexes, so it only
needs to fetch infos from repo and build a chunk index once per backup
archive.
If out of sync, missing archive indexes get added, outdated indexes
get removed and a new master chunks index is built by merging all
archive indexes.
r� r re c s t | �}tj�� || �}|S r? )r r0 r1 r2 )r� r� Zid_hexr1 )� archive_pathr7 r8 �mkpath� s z LocalCache.sync.<locals>.mkpathc s6 �j r,t�� �} dd� | D �dd� | D �B S t� S d S )Nc S s h | ]}t |�d krt|��qS )�@ )r� r r� r7 r7 r8 � <setcomp>� r� z;LocalCache.sync.<locals>.cached_archives.<locals>.<setcomp>c S s2 h | ]*}t |�d kr|�d�rt|dd� ��qS )�H �.compactNr )r� �endswithr r� r7 r7 r8 r � r� )�do_cacher0 r� r� )Zfns�r r6 r7 r8 �cached_archives� s
�z(LocalCache.sync.<locals>.cached_archivesc s dd� � j j�� D �S )Nc S s h | ]
}|j �qS r7 )r� )r@ �infor7 r7 r8 r � r� z9LocalCache.sync.<locals>.repo_archives.<locals>.<setcomp>)rW r� �listr7 rD r7 r8 �
repo_archives� s z&LocalCache.sync.<locals>.repo_archivesc s | D ] }� |� qd S r? r7 )�idsr� )�cleanup_cached_archiver7 r8 �cleanup_outdated� s z)LocalCache.sync.<locals>.cleanup_outdatedTc s� z$t �� | �� t �� | �d � W n ty6 Y n0 |s@d S z,t �� | dd�� t �� | dd�d � W n ty~ Y n0 d S )Nz
.integrityr r� )r0 �unlinkr] )r� �cleanup_compact)r r7 r8 r! � s z/LocalCache.sync.<locals>.cleanup_cached_archivec s� | � � }g }t� �rT|D ]8}� �|�}|rF| | j|jd�}|| |< q|�|� qn|}t|�j�|��D ]@\}}| | jt|�d�}|| |< |� |< �d7 ��t|�7 �qjdS )z�
Archives created with AdHocCache will have csize=0 in all chunk list entries whose
chunks were already in the repository.
Scan *chunk_idx* for entries where csize=0 and fill in the correct information.
)r� r N) Zzero_csize_idsr� r� r� r� �append�zipr, �get_many)� chunk_idxZall_missing_idsZ fetch_ids�id_Zalready_fetched_entryr� r� )�chunks_fetched_size_index�decrypted_repository�fetched_bytes_for_csize�fetched_chunks_for_csizer7 r8 �fetch_missing_csize� s
z,LocalCache.sync.<locals>.fetch_missing_csizec
s� |� | �\}}|�| dt|�|� �jj|dd�\}}}t|d�}|jdkrTtd��t|�}t |j
|� |j
��D ]>\} \}}|�| dt|�|� �t|�7 ��d7 �|� |� qp�j
rȈ |� �| |� d S )Nr T)Zforce_tam_not_required)Z
internal_dictz Unknown archive metadata version)r� �addr� rM Zunpack_and_verify_archiver r� r� r r& r� r' r� r )
�
archive_idr+ r( r� r� r� Zverified�_r� Zitem_id)r. � processed_item_metadata_bytes� processed_item_metadata_chunksr6 �write_archive_indexr7 r8 �fetch_and_build_idx� s
z,LocalCache.sync.<locals>.fetch_and_build_idxc s� � |� � 7 � �| dd�}�| dd�}zDt|dt| �d d��}|�|� W d � n1 s\0 Y W n ty� t|� Y n0 t�||� d S )Nr r� z.tmpT)r1 rV �filename)Zcompactr# r rV r� r r0 r )r0 r( r� Zfn_tmprN )�"compact_chunks_archive_saved_spacer r7 r8 r4 s
�, z,LocalCache.sync.<locals>.write_archive_indexc s �| �}t �d|� z�zRt|d dd�� }tj|dd�}W d � n1 sL0 Y � | dd� |W W S ty� t|dd��}t�|�}W d � n1 s�0 Y Y n0 W nF t� y� } z,t �d||� � | � tt � W Y d }~d S d }~0 0 t �
d |� � | � �| |� |S )
Nz-Reading cached archive chunk index for %s ...r Fr T)Zpermit_compact)r$ z1Cached archive chunk index of %s is corrupted: %sz6Found non-compact index for %s, converting to compact.) rK r r# r rG r] r$ �errorr r rR )r0 r� Zarchive_chunk_idx_pathrN �archive_chunk_idxr )r! r r4 r7 r8 �read_archive_index s( , 2
z+LocalCache.sync.<locals>.read_archive_indexc s0 i }� j j�� D ]}|j| v r|j||j< q|S r? )rW r� r r� �name)� archive_idsZ
archive_namesr rD r7 r8 �get_archive_ids_to_names/ s
z1LocalCache.sync.<locals>.get_archive_ids_to_namesc s� t �d� � � }� � }t �dt|�t|�t|| �t|| �� | �� �|| � t�
j�}|�r��
jsnd nt|d�} tt|�dddd�}�|�}|�� D ]�\}}|j t
|�gd� �
j�r ||v r� ||�}|d u r�|� |� ||v�r
t �d |� t� }�|�|� t �d
� | � |� q�| �p.t|d�} t �d |� �|�| � q��
j�sZ�| � |�
� t �d t���� t �d
t� ��
� t �dt��� t �d� | S )Nz Synchronizing chunks cache...zIArchives: %d, w/ cached Idx: %d, w/ outdated Idx: %d, w/o cached Idx: %d.�Zusableg�������?z3%3.0f%% Syncing chunks cache. Processing archive %sz
cache.sync)�total�steprc r )r z.Fetching and building archive index for %s ...z$Merging into master chunks index ...z!Fetching archive index for %s ...zlCache sync: had to fetch %s (%d chunks) because no archive had a csize set for them (due to --no-cache-sync)z0Cache sync: processed %s (%d chunks) of metadataz;Cache sync: compact chunks.archive.d storage saved %s byteszDone.)rK r r� �clearr, r r r r� �showr r� �merger rR r ) r( Z
cached_idsr< Zmaster_index_capacityr
Zarchive_ids_to_namesr0 r� r9 )r r" r7 r+ r5 r. r, r- r= r2 r3 r: r r6 r7 r8 �create_master_idx; s^
�
�
���
z*LocalCache.sync.<locals>.create_master_idxc sp zt �t j��jd�� W n Y n0 zt �t j��jd�� W n Y n0 zt �� � W n Y n0 dS )z?bring old cache dirs into the desired state (cleanup and adapt)zchunks.archivezchunks.archive.tmpN)r0 r# r1 r2 r r7 r r7 r8 �legacy_cleanupn s z'LocalCache.sync.<locals>.legacy_cleanup)Zdecrypted_cacheN)re )T)r0 r1 r2 r rW Z check_repository_compatibilityr Z OperationZREADr r' r, rM r� r r� )r6 rD rE r7 )r r r* r! r" r7 r+ r5 r. r, r- r= r r2 r3 r: r r6 r4 r8 r� � s2
! &3zLocalCache.syncc C s* t j}| jj|@ rdS | jj|ks&dS dS )NFT)r �SUPPORTED_REPO_FEATURESrb r� r� )r6 � my_featuresr7 r7 r8 r� � s z$LocalCache.check_cache_compatibilityc C s� t �d� tj�| jd�}tj�|�rRt�tj�| jd�� t�tj�| jd�� t � | _
t tj�| jt � �dd�� W d � n1 s�0 Y d| j
_| j
j�ddd� t� | j
_t� | j
_d S )Nz9Discarding incompatible cache and forcing a cache rebuildr� Tr� re r� rW )rK rL r0 r1 r2 r� r; r< r� r r� r& r� rb r� r� r� r� r� )r6 r r7 r7 r8 r� � s
zLocalCache.wipe_cachec C sZ | j �� }tj}t� }|�� D ]\}}|�|� q | jj�|| � | jj �||@ � d S r? )
rW Zget_all_mandatory_featuresr rF r� r� �updaterb r� r� )r6 Zoperation_to_features_maprG Z
repo_featuresZ operationZfeaturesr7 r7 r8 r� � s
z LocalCache.update_compatibilityc
C s� | j s| �� t|�}| �||�}|r6|s6| �||�S | j�|�}t|�} | jj|||d� | j �
|d|| � |� || | � t ||| �S )N��waitr )
r� r r� �
seen_chunk� chunk_increfrM �encryptr, �putr� r/ rH r �
r6 r� �chunkr� Z overwriterJ �size�refcountr� r� r7 r7 r8 � add_chunk� s zLocalCache.add_chunkc C sJ | j �|tdd d ��\}}}|d urF|d urF||krFtd|||f ��|S )Nr z@chunk has same id [%r], but different size (stored: %d new: %d)!)r� r� r
r� )r6 r� rQ rR Z stored_sizer1 r7 r7 r8 rK � s �zLocalCache.seen_chunkc C s> | j s| �� | j�|�\}}}|j||d|d� t|||�S �NF��part)r� r r� �increfrH r �r6 r� r� rQ rV �count�_sizer� r7 r7 r8 rL � s
zLocalCache.chunk_increfc C sr | j s| �� | j�|�\}}}|dkrX| j|= | jj||d� |j| | d|d� n|j| | d|d� d S �Nr rI TrU F)r� r r� �decrefr, �deleterH �r6 r� r� rJ rV rY rQ r� r7 r7 r8 � chunk_decref� s zLocalCache.chunk_decrefc C s2 t �|j�sdS | j}d|v r,t�d� dS d|v rBt�d� dS | j�|�}|sbt�d|� dS tt �
|�� }d|v r�|j |j kr�t�d|� d S d
|v r�|j
|jkr�t�d |� d S d |v r�t|j�|jkr�t�d
|� d S d|v �r t|j�|jk�r t�d|� d S t �|j|jdd��| j|< d|jfS )aj
Check if we know the file that has this path_hash (know == it is in our files cache) and
whether it is unchanged (the size/inode number/cmtime is same for stuff we check in this cache_mode).
:param hashed_path: the file's path as we gave it to hash(hashed_path)
:param path_hash: hash(hashed_path), to save some memory in the files cache
:param st: the file's stat() result
:return: known, ids (known is True if we have infos about this file in the cache,
ids is the list of chunk ids IF the file has not changed, otherwise None).
�FNr� z UNKNOWN: files cache disabled�rz UNKNOWN: rechunking enforcedz*UNKNOWN: no file metadata in cache for: %r�sz(KNOWN-CHANGED: file size has changed: %r)TN�iz0KNOWN-CHANGED: file inode number has changed: %r�cz)KNOWN-CHANGED: file ctime has changed: %r�mz)KNOWN-CHANGED: file mtime has changed: %rr )�inoder� T)�stat�S_ISREG�st_moder� r� rR r� r� r) r r rQ �st_sizerf �st_inor r � st_ctime_ns� st_mtime_nsr� r� � chunk_ids)r6 � hashed_pathr� �str� r� r7 r7 r8 �file_known_and_unchanged� s8
z#LocalCache.file_known_and_unchangedc C s� t �|j�sd S | j}d|v r,t�d� d S d|v rDd}t|j�}n&d|v r\d}t|j�}nd}t|j�}t d|j
|j t |�|d�}t
�|�| j|< t| jp�d|�| _t�d |jd
t|j� d �||� d S ) Nr� z*FILES-CACHE-NOUPDATE: files cache disabledrd �ctimere �mtimer )r� rf rQ r rn z)FILES-CACHE-UPDATE: put %r [has %s] <- %rz [%d entries])rn )rg rh ri r� r� rR r rl rm r) rk rj r r r� r� rf r� r� r� rn ) r6 ro r� rp r r� Z cmtime_typeZ cmtime_nsr� r7 r7 r8 �
memorize_file s*
�zLocalCache.memorize_file)FT)N)NF)TF)rv rw rx ry r r9 r� r� r� r� rF r� r� r r� r� r� r� r� r� rS rK rL r_ rq rt r7 r7 r7 r8 r� � s4 �
,!) s
4r� c @ s� e Zd ZdZdZd dd�Zdd � Zd
d � ZdZd Z d
d� Z
dd� Z d dd�Z d!dd�Z
d"dd�Zd#dd�Zdd� Zdd � Zd d � ZdS )$r� a�
Ad-hoc, non-persistent cache.
Compared to the standard LocalCache the AdHocCache does not maintain accurate reference count,
nor does it provide a files cache (which would require persistence). Chunks that were not added
during the current AdHocCache lifetime won't have correct size/csize set (0 bytes) and will
have an infinite reference count (MAX_VALUE).
z�All archives: unknown unknown unknown
Unique chunks Total chunks
Chunk index: {0.total_unique_chunks:20d} unknownTNFc C s\ t j| |d� || _|| _|| _|| _d| _t|�| _| jj |||d� i | _
t � d� d S )Nr� Frp z1Note: --no-cache-sync is an experimental feature.)
r� r9 r, rM rW r� � _txn_activer* r� rt r� rK rL )r6 r, rM rW rn ro r� r� r7 r7 r8 r9 5 s
zAdHocCache.__init__c C s | S r? r7 rD r7 r7 r8 r� F s zAdHocCache.__enter__c C s d S r? r7 r� r7 r7 r8 r� I s zAdHocCache.__exit__r� c C s t �d� dS )Nz$UNKNOWN: files cache not implementedr` )r� rR )r6 ro r� rp r7 r7 r8 rq O s
z#AdHocCache.file_known_and_unchangedc C s d S r? r7 )r6 ro r� rp r r7 r7 r8 rt S s zAdHocCache.memorize_filec
C s� | j s| �� t|�}| �||�}|r6| j|||d�S | j�|�}t|�} | jj|||d� | j �
|d|| � |� || | � t ||| �S )N�rQ rI r )
ru r r� rK rL rM rM r, rN r� r/ rH r rO r7 r7 r8 rS V s zAdHocCache.add_chunkc C sL | j s| �� | j�|tdd d ��}|jrF|rF|jsF|j|d�| j|< |jS )Nr rv )ru r r� r� r
rR rQ r� )r6 r� rQ r� r7 r7 r8 rK e s zAdHocCache.seen_chunkc C sF | j s| �� | j�|�\}}}|p&|}|j||d|d� t|||�S rT )ru r r� rW rH r rX r7 r7 r8 rL p s zAdHocCache.chunk_increfc C sr | j s| �� | j�|�\}}}|dkrX| j|= | jj||d� |j| | d|d� n|j| | d|d� d S r[ )ru r r� r\ r, r] rH r^ r7 r7 r8 r_ { s zAdHocCache.chunk_decrefc C s& | j s
d S | j�| j| j� d| _ d S �NF)ru r� rX rW rM rD r7 r7 r8 r� � s zAdHocCache.commitc C s d| _ | `d S rw )ru r� rD r7 r7 r8 r� � s zAdHocCache.rollbackc
C s� d| _ t| j�}t|d d�| _t|ddd�}t� }d}d }| jjt|d�}|d 7 }|sZq�|j t|�d
� |d }t
tj ddd �}|D ]}|| j|< q�q<| j| j j
= t� | p�d
} |�� t�d|| |t|d | �� d S )NTg�������?r> z!Downloading chunk list... %3.0f%%zcache.download_chunks)r? rc r r )�limit�markerr )Zincrease���)rR rQ r� g{�G�z�?zBAdHocCache: downloaded %d chunk IDs in %.2f s (%d requests), ~%s/s�" )ru r� r, r r� r r r r( rB r
Z MAX_VALUErW Z MANIFEST_IDr rK rR r )
r6 Z
num_chunksr
Zt0Z num_requestsry �resultZ
init_entryr) Zdurationr7 r7 r8 r � s0
� �zAdHocCache.begin_txn)TNFF)FT)N)NF)TF)rv rw rx ry r� r9 r� r� r� r� rq rt rS rK rL r_ r� r� r r7 r7 r7 r8 r� % s$ �
r� )N)Fr� r� r0 r; rg Zbinasciir � collectionsr �timer rK r r� Z constantsr r Z hashindexr r
r Zhelpersr r
r r r r r r r r r r r r r r r r r r� r r Z
crypto.keyr! Zcrypto.file_integrityr" r# r$ Zlockingr% �platformr& Zremoter' r, r( r) r* rt r� r/ r� r� rs r` r� r� r� r7 r7 r7 r8 �<module> sb
%
dA0 c