SillyTavern Tutorials / Troubleshooting / OogaBooga Text Generation Web UI
Resolving the 'LlamaCppModel' Object Missing 'Device' Attribute Error in Text Generation Web UI - Ubuntu Linux
I recently ran into an issue while using the OogaBooga Text Generation Web UI with some GGUF models. The models loaded fine into Text Gen Web UI, but when I tried to use the OpenAI API extension for text completions via SillyTavern, I got an error from the OpenAI extension in the LogitsBiasProcessor function saying the LlamaCppModel object had no attribute device. After some troubleshooting, I figured out how to fix it. This may be hyperspecific to my hardware and software version but I did not find any other information about this on the web so I figured I'd make a post about it.
My Setup/Environment
I have recently built an AI server on a Dell Poweredge R730, it is loaded up with Proxmox to manage the VMs following magiccodingman's Budget-AI-Workstation-Build on github.
Here’s a quick look at my environment details that this fix worked on:
- Operating System: Ubuntu 22.04.04 LTS VM
- Processor: 2 X Intel® Xeon(R) CPU E5-2690 v4 @ 2.60GHz × 14 core
- Graphics: Tesla P40/PCIe/SSE2 / Tesla P40/PCIe/SSE2
- Nvidia Driver Version: 555.42.02
- CUDA Version: 12.5
- Text-Web-UI Version: 1.8
- Model I was trying to load: TheBloke / nous-capybara-limarpv3-34b.Q4_K_M-gguf
- Model Loader: llama.cpp
- Model Settings (click to open larger):
The Error
Here's the error message I was getting when I tried to use the OpenAI API for text completions inside SillyTavern, the console would print the following error:
Exception in ASGI application
Traceback (most recent call last):
File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/sse_starlette/sse.py", line 247, in __call__
await wrap(partial(self.listen_for_disconnect, receive))
File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/sse_starlette/sse.py", line 236, in wrap
await func()
File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/sse_starlette/sse.py", line 191, in listen_for_disconnect
message = await receive()
File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
await self.message_event.wait()
File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/asyncio/locks.py", line 213, in wait
await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope
During handling of the above exception, another exception occurred:
+ Exception Group Traceback (most recent call last):
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
| return await self.app(scope, receive, send)
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
| await super().__call__(scope, receive, send)
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
| raise exc
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
| await self.app(scope, receive, _send)
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
| await self.app(scope, receive, send)
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
| raise exc
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| await app(scope, receive, sender)
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
| await route.handle(scope, receive, send)
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
| await self.app(scope, receive, send)
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
| raise exc
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| await app(scope, receive, sender)
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
| await response(scope, receive, send)
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/sse_starlette/sse.py", line 233, in __call__
| async with anyio.create_task_group() as task_group:
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/sse_starlette/sse.py", line 236, in wrap
| await func()
| File "/path/to/text-generation-webui/installer_files/env/lib/python3.11/site-packages/sse_starlette/sse.py", line 221, in stream_response
| async for data in self.body_iterator:
| File "/path/to/text-generation-webui/extensions/openai/script.py", line 106, in generator
| for resp in response:
| File "/path/to/text-generation-webui/extensions/openai/completions.py", line 559, in stream_completions
| for resp in completions_common(body, is_legacy, stream=True):
| File "/path/to/text-generation-webui/extensions/openai/completions.py", line 402, in completions_common
| generate_params = process_parameters(body, is_legacy=is_legacy)
| File "/path/to/text-generation-webui/extensions/openai/completions.py", line 113, in process_parameters
| logits_processor = [LogitsBiasProcessor(logit_bias)]
| File "/path/to/text-generation-webui/extensions/openai/completions.py", line 39, in __init__
| self.values = torch.tensor(values, dtype=torch.float, device=shared.model.device)
| AttributeError: 'LlamaCppModel' object has no attribute 'device'
+------------------------------------
Steps to Fix It
Here’s what I did to fix the issue:
- Update the Repository
- Install or Update Required Packages
- Modify the Code to Handle Missing Attributes
- Restart the Server
Detailed Steps:
- Update the Repository
- Navigate to the Text Generation Web UI directory:
cd/path/to/text-generation-webui- Run the linux update script
./update_wizard_linux.sh- Install or Update Required Packages
- Install or update the required Python packages:
pip install -r requirements.txt --upgrade- Modify the OpenAI Code to Handle Missing Attributes
- Navigate to the directory containing the
completions.pyfile: cd/path/to/text-generation-webui/extensions/openai- Open the
completions.pyfile in a text editor: nano completions.py- Find the the following line (around line 39)
self.values = torch.tensor(values, dtype=torch.float, device=shared.model.device)- Replace that line with the following, ensure the Python indentation is consistent with the surrounding functions indent or it will break the extension.
ifhasattr(shared.model, 'device'):
self.values = torch.tensor(values, dtype=torch.float, device=shared.model.device)
else:
self.values = torch.tensor(values, dtype=torch.float)
- Save and Close the File
4. Restart the Server
- Finally, I restarted the server to apply the changes:
cd/path/to/text-generation-webui/- Start the Text Web UI server:
./start_linux.sh
Conclusion
And that’s it! By following these steps, I was able to resolve the issue with the missing device attribute in the LlamaCppModel object when loading certain GGUF models. Hopefully, this helps you out if you run into the same problem.
Appendix
This section is to help users find this guide if they are encountering the same issue, you can disregard this section. If you're encountering an error with the LlamaCppModel object missing the device attribute while using the Text Generation Web UI (TWUI) with GGUF models and trying to run the OpenAI API for text completions, you might see messages like "AttributeError: 'LlamaCppModel' object has no attribute 'device'" or issues related to loading models and running text completions. This guide covers fixing common problems such as tensor device attribute errors, resolving sse_starlette asyncio errors, handling Uvicorn ASGI application errors, and addressing GPU model loading issues in TWUI. Follow these steps to ensure your models load correctly and the API functions smoothly.