added thread limit option to database
created proper README
This commit is contained in:
parent
c3405c42cc
commit
d12377e9c5
90
README.md
90
README.md
|
@ -1,12 +1,82 @@
|
||||||
# Programmentwurf
|
# Imsearch
|
||||||
|
|
||||||
Die Beschreibung der Aufgabenstellung ist unter [Programmentwurf.md](https://github.com/programmieren-mit-rust/programmentwurf/blob/main/Programmentwurf.md) zu finden. Diese `Readme.md` ist durch etwas Sinnvolles zu ersetzen.
|
Extensible library for creating an image based search engine. The library exposes the functionality to create databases which index various images stored as png files.
|
||||||
|
|
||||||
# WICHTIG!
|
Files can be compared for similarity by either premade features or custom ones. The basic idea of handling the library is as follows:
|
||||||
Kleiner reminder, wenn ihr Sachen pushed in das repo, die eurer Anischt nach fertig sind (z.B für einen Pull-Request!), bitte mit den folgenden Commands auf Fehler/Warnings überprüfen:
|
|
||||||
- `cargo fmt` für formattierung
|
- Create a new database
|
||||||
- `cargo clippy` für warnings
|
- Add some features to the database
|
||||||
- `cargo test doc` für documentation tests
|
- Add some images to the database
|
||||||
optional:
|
- Search for some images in the database by a certain feature
|
||||||
- `cargo test` für module tests
|
- Save the database to disk
|
||||||
- `cargo bench` für benchmarks
|
|
||||||
|
or:
|
||||||
|
|
||||||
|
- Load a database form disk
|
||||||
|
- Supply generator functions
|
||||||
|
- Search for some images in the database by a certain feature
|
||||||
|
- Add some images to the database
|
||||||
|
- Save the database to disk
|
||||||
|
|
||||||
|
# Examples:
|
||||||
|
## Define a new feature
|
||||||
|
```rust
|
||||||
|
/// Compute the average value of the tree color channels of a given image
|
||||||
|
fn average_rgb_value(image: Arc<Image<f32>>) -> (String, FeatureResult) {
|
||||||
|
let bright = image
|
||||||
|
.pixels()
|
||||||
|
.iter()
|
||||||
|
.map(|(r, g, b, _)| (r + g + b) / 3.0 / 255.0)
|
||||||
|
.sum::<f32>();
|
||||||
|
|
||||||
|
(
|
||||||
|
String::from("average_brightness"),
|
||||||
|
FeatureResult::Percent(bright / image.pixels().len() as f32),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
## Create a new database
|
||||||
|
```rust
|
||||||
|
let files: Vec<PathBuf> = std::fs::read_dir("image/folder/")
|
||||||
|
.unwrap()
|
||||||
|
.map(|f| f.unwrap().path())
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
let feats: Vec<FeatureGenerator> = vec![average_rgb_value];
|
||||||
|
|
||||||
|
let db = Database::new(&files, feats).unwrap();
|
||||||
|
|
||||||
|
db.write_to_file(json);
|
||||||
|
```
|
||||||
|
## Read a new database and search for similar images
|
||||||
|
```rust
|
||||||
|
let db = Database::from_file(Path::new("db.json"));
|
||||||
|
|
||||||
|
for results in db
|
||||||
|
.search(
|
||||||
|
std::path::Path::new("path/to/image.png"),
|
||||||
|
average_brightness,
|
||||||
|
)
|
||||||
|
.unwrap()
|
||||||
|
{
|
||||||
|
println!(
|
||||||
|
"path: {} similarity: {}",
|
||||||
|
results.0.as_os_str().to_str().unwrap(),
|
||||||
|
results.1
|
||||||
|
);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
# Details
|
||||||
|
Processing of features for images are multithreaded. Features that are calculated for images only get their results stored. The generator function used to calculate won't get serialized. This implies that
|
||||||
|
in order to compute the features for images the generator functions have to be passed to the database
|
||||||
|
after it has been read from a file.
|
||||||
|
## Limiting thread usage
|
||||||
|
You can limit the number of threads to be used by calling `set_limit()` on the database.
|
||||||
|
Note that the thread pool will automatically try to detect the optimal number of threads to use.
|
||||||
|
As long as no edge case such as running in an over committed virtual machine applies this will be
|
||||||
|
good enough for most cases.
|
||||||
|
## Image formats
|
||||||
|
The library can only handle png files through the `png` crate. Note that not all colortypes are supported. Due to the poor capabilites of the crate pngs with indexed palettes are not functional
|
||||||
|
will cause functions to return error values.
|
||||||
|
## Memory usage
|
||||||
|
The database won't hold all images in ram at the same time. They are loaded on demand when calculating features for them. This may cause increased disk usage but will prevent ram overcommitment.
|
|
@ -174,6 +174,13 @@ where
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Set a new limit to the amount of used threads.
|
||||||
|
/// This will only take effect when new threads are created. Currently running threads
|
||||||
|
/// will continue to do so.
|
||||||
|
pub fn set_limt(&mut self, limit: NonZeroUsize) {
|
||||||
|
self.limit = limit;
|
||||||
|
}
|
||||||
|
|
||||||
/// Put a new job into the queue to be executed by a thread in the future.
|
/// Put a new job into the queue to be executed by a thread in the future.
|
||||||
/// The priority of the job will determine if the job will be put at the start or end of the queue.
|
/// The priority of the job will determine if the job will be put at the start or end of the queue.
|
||||||
/// See [`crate::multithreading::Priority`].
|
/// See [`crate::multithreading::Priority`].
|
||||||
|
|
|
@ -33,6 +33,7 @@ use serde::{Deserialize, Serialize};
|
||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
use std::default::Default;
|
use std::default::Default;
|
||||||
use std::fs;
|
use std::fs;
|
||||||
|
use std::num::NonZeroUsize;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
@ -291,6 +292,17 @@ impl Database {
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Limit the amount of threads to use when processing features
|
||||||
|
pub fn limit_thread_usage(&mut self, limit: NonZeroUsize) {
|
||||||
|
self.threadpool.set_limt(limit);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Register the supplied generators.
|
||||||
|
/// All previously registered generators will be replaced with the new generators
|
||||||
|
pub fn register_generators(&mut self, generators: Vec<FeatureGenerator>) {
|
||||||
|
self.generators = generators;
|
||||||
|
}
|
||||||
|
|
||||||
/// with add_image you can add images in a existing database.
|
/// with add_image you can add images in a existing database.
|
||||||
/// databases from a file are read only.
|
/// databases from a file are read only.
|
||||||
pub fn add_image(&mut self, path: &Path) -> Result<(), &'static str> {
|
pub fn add_image(&mut self, path: &Path) -> Result<(), &'static str> {
|
||||||
|
|
Loading…
Reference in New Issue